[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:08:27 dist-test-slave-tm1g krb5kdc[17539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:08:27 dist-test-slave-tm1g krb5kdc[17539](info): set up 2 sockets
Aug 07 03:08:27 dist-test-slave-tm1g krb5kdc[17539](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260807 03:08:28.176419 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:08:28 dist-test-slave-tm1g krb5kdc[17539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:08:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:08:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:08:28.360860 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:35035
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:35035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:28.717080 17552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:28.717412 17552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:28.717515 17552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:28.724749 17552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:08:28.724841 17552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:28.724888 17552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:28.724944 17552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:08:28.724987 17552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:08:28.735031 17552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:35035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:35035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:28.736748 17552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:28.738574 17552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:28.756160 17558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:28.756127 17557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:28.756250 17560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:28.757898 17552 server_base.cc:1034] running on GCE node
I20260807 03:08:28.759140 17552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:28.761402 17552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:28.762651 17552 hybrid_clock.cc:648] HybridClock initialized: now 1786072108762553 us; error 82 us; skew 500 ppm
I20260807 03:08:28.763164 17552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:28 dist-test-slave-tm1g krb5kdc[17539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072108, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:28.766546 17552 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:08:28.768652 17552 webserver.cc:460] Webserver started at http://127.17.31.62:40901/ using document root <none> and password file <none>
I20260807 03:08:28.769698 17552 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:28.769850 17552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:28.770396 17552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:28.774113 17552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "e9218c91c8d44645bd90915f8e0fe93e"
format_stamp: "Formatted at 2026-08-07 03:08:28 on dist-test-slave-tm1g"
server_key: "c9e3250c65600fbe78e9ecd96bb14a48"
server_key_iv: "b622c891d14627600a330e0c944df288"
server_key_version: "encryptionkey@0"
I20260807 03:08:28.774960 17552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "e9218c91c8d44645bd90915f8e0fe93e"
format_stamp: "Formatted at 2026-08-07 03:08:28 on dist-test-slave-tm1g"
server_key: "c9e3250c65600fbe78e9ecd96bb14a48"
server_key_iv: "b622c891d14627600a330e0c944df288"
server_key_version: "encryptionkey@0"
I20260807 03:08:28.782277 17552 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260807 03:08:28.787356 17567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:28.789147 17552 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 03:08:28.789423 17552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "e9218c91c8d44645bd90915f8e0fe93e"
format_stamp: "Formatted at 2026-08-07 03:08:28 on dist-test-slave-tm1g"
server_key: "c9e3250c65600fbe78e9ecd96bb14a48"
server_key_iv: "b622c891d14627600a330e0c944df288"
server_key_version: "encryptionkey@0"
I20260807 03:08:28.789621 17552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:28.824581 17552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:28.829161 17552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:28.829532 17552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:28.851526 17552 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:35035
I20260807 03:08:28.851570 17620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:35035 every 8 connection(s)
I20260807 03:08:28.854065 17552 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:08:28.858296 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 17552
I20260807 03:08:28.858777 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:08:28.862603 17532 external_mini_cluster.cc:1428] Setting key e3c90f264f4a259452c3c6f3419b6062
I20260807 03:08:28.863042 17621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:28.879602 17621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e: Bootstrap starting.
I20260807 03:08:28.886780 17621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:28.888216 17621 log.cc:824] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e: Log is configured to *not* fsync() on all Append() calls
I20260807 03:08:28.892462 17621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e: No bootstrap required, opened a new log
Aug 07 03:08:28 dist-test-slave-tm1g krb5kdc[17539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:28.902258 17621 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9218c91c8d44645bd90915f8e0fe93e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35035 } }
I20260807 03:08:28.902839 17621 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:28.903069 17621 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9218c91c8d44645bd90915f8e0fe93e, State: Initialized, Role: FOLLOWER
I20260807 03:08:28.904371 17621 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9218c91c8d44645bd90915f8e0fe93e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35035 } }
I20260807 03:08:28.904817 17621 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:08:28.905100 17621 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:08:28.905359 17621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:28.908485 17621 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9218c91c8d44645bd90915f8e0fe93e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35035 } }
I20260807 03:08:28.909557 17621 leader_election.cc:302] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9218c91c8d44645bd90915f8e0fe93e; no voters:
I20260807 03:08:28.910521 17621 leader_election.cc:288] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:08:28.910703 17626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:08:28.912074 17623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:28.870130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55096 (local address 127.17.31.62:35035)
0807 03:08:28.871215 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:28.871230 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:28.874145 (+ 2915us) server_negotiation.cc:407] Connection header received
0807 03:08:28.884675 (+ 10530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:28.884737 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:28.885440 (+ 703us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:28.886007 (+ 567us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:28.889327 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:28.891494 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:28.893270 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:28.894717 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:28.894980 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:28.900485 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:28.900559 (+ 74us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:28.900600 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:28.900645 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:28.905374 (+ 4729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:28.906241 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:28.906249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:28.906256 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:28.906542 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:28.907003 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:28.907009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:28.907012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:28.907904 (+ 892us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:28.908291 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:28.909252 (+ 961us) server_negotiation.cc:299] Negotiation successful
0807 03:08:28.909746 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":375,"threads_started":1}
I20260807 03:08:28.913938 17621 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:08:28.914295 17626 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [term 1 LEADER]: Becoming Leader. State: Replica: e9218c91c8d44645bd90915f8e0fe93e, State: Running, Role: LEADER
I20260807 03:08:28.914994 17626 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9218c91c8d44645bd90915f8e0fe93e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35035 } }
I20260807 03:08:28.922004 17628 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9218c91c8d44645bd90915f8e0fe93e. Latest consensus state: current_term: 1 leader_uuid: "e9218c91c8d44645bd90915f8e0fe93e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9218c91c8d44645bd90915f8e0fe93e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35035 } } }
I20260807 03:08:28.921958 17627 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9218c91c8d44645bd90915f8e0fe93e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9218c91c8d44645bd90915f8e0fe93e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35035 } } }
I20260807 03:08:28.922470 17627 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [sys.catalog]: This master's current role is: LEADER
I20260807 03:08:28.922470 17628 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e [sys.catalog]: This master's current role is: LEADER
I20260807 03:08:28.925195 17633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:08:28.932085 17633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:08:28.946599 17633 catalog_manager.cc:1347] Generated new cluster ID: ee7276f99aa747b1a7d620fdd8f71875
I20260807 03:08:28.946746 17633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:08:28.978823 17633 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:08:28.980242 17633 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:08:28.989987 17633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9218c91c8d44645bd90915f8e0fe93e: Generated new TSK 0
I20260807 03:08:28.991051 17633 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:08:29.046711 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35035
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:29.375505 17647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:29.375809 17647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:29.375988 17647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:29.382987 17647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:29.383126 17647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:29.383304 17647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:08:29.392752 17647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:29.394066 17647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:29.395450 17647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:29.408815 17653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:29.408790 17654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:29.408859 17656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:29.409585 17647 server_base.cc:1034] running on GCE node
I20260807 03:08:29.410112 17647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:29.411001 17647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:29.412199 17647 hybrid_clock.cc:648] HybridClock initialized: now 1786072109412164 us; error 28 us; skew 500 ppm
I20260807 03:08:29.412495 17647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:29 dist-test-slave-tm1g krb5kdc[17539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072109, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:29.415696 17647 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:08:29.416981 17647 webserver.cc:460] Webserver started at http://127.17.31.1:44077/ using document root <none> and password file <none>
I20260807 03:08:29.417572 17647 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:29.417680 17647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:29.417929 17647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:29.420679 17647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "2de0c62196e442ed80b76f39cf5f251a"
format_stamp: "Formatted at 2026-08-07 03:08:29 on dist-test-slave-tm1g"
server_key: "0471529a6c3b6c9268ab654efcb5099d"
server_key_iv: "d01fe46931f2dc1f8a4d13a1d71bf35a"
server_key_version: "encryptionkey@0"
I20260807 03:08:29.421365 17647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "2de0c62196e442ed80b76f39cf5f251a"
format_stamp: "Formatted at 2026-08-07 03:08:29 on dist-test-slave-tm1g"
server_key: "0471529a6c3b6c9268ab654efcb5099d"
server_key_iv: "d01fe46931f2dc1f8a4d13a1d71bf35a"
server_key_version: "encryptionkey@0"
I20260807 03:08:29.426175 17647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260807 03:08:29.429416 17663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:29.430663 17647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:08:29.430830 17647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "2de0c62196e442ed80b76f39cf5f251a"
format_stamp: "Formatted at 2026-08-07 03:08:29 on dist-test-slave-tm1g"
server_key: "0471529a6c3b6c9268ab654efcb5099d"
server_key_iv: "d01fe46931f2dc1f8a4d13a1d71bf35a"
server_key_version: "encryptionkey@0"
I20260807 03:08:29.431026 17647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:29.453155 17647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:29.456794 17647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:29.457085 17647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:29.458109 17647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:08:29.459640 17647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:29.459715 17647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:29.459946 17647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:29.460016 17647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:29.495438 17647 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:42853
I20260807 03:08:29.495483 17777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:42853 every 8 connection(s)
I20260807 03:08:29.496696 17647 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:08:29.503269 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 17647
I20260807 03:08:29.503511 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:08:29.504364 17532 external_mini_cluster.cc:1428] Setting key 2e5b78b0461146b842814f64d69f23b7
Aug 07 03:08:29 dist-test-slave-tm1g krb5kdc[17539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072109, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:29.518956 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:29.499666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:37481 (local address 127.17.31.62:35035)
0807 03:08:29.500077 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:29.500081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:29.501051 (+ 970us) server_negotiation.cc:407] Connection header received
0807 03:08:29.502926 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:29.502930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:29.503008 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:29.503099 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:29.505154 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:29.505978 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:29.508284 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:29.508681 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:29.508775 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:29.512924 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:29.512929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:29.512932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:29.512962 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:29.515805 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:29.516698 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:29.516702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:29.516704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:29.516815 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:29.517287 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:29.517290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:29.517292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:29.517527 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:29.517630 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:29.518556 (+ 926us) server_negotiation.cc:299] Negotiation successful
0807 03:08:29.518705 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":225,"threads_started":1}
I20260807 03:08:29.520437 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:29.500229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35035 (local address 127.17.31.1:37481)
0807 03:08:29.500892 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:29.500929 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:29.502647 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:29.503371 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:29.503385 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:29.504146 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:29.504968 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:29.504992 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:29.506142 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:29.506148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:29.508140 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:29.508158 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:29.508908 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:29.508919 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:29.509258 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:29.509985 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:29.510017 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:29.512726 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:29.515936 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:29.515947 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:29.515965 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:29.516587 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:29.516924 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:29.516931 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:29.516937 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:29.517181 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:29.517652 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:29.517662 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:29.518091 (+ 429us) client_negotiation.cc:769] Sending connection context
0807 03:08:29.518467 (+ 376us) client_negotiation.cc:240] Negotiation successful
0807 03:08:29.518984 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":261,"threads_started":1}
I20260807 03:08:29.522054 17778 heartbeater.cc:347] Connected to a master server at 127.17.31.62:35035
I20260807 03:08:29.522544 17778 heartbeater.cc:464] Registering TS with master...
I20260807 03:08:29.523329 17778 heartbeater.cc:511] Master 127.17.31.62:35035 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
I20260807 03:08:29.527258 17585 ts_manager.cc:194] Registered new tserver with Master: 2de0c62196e442ed80b76f39cf5f251a (127.17.31.1:42853)
I20260807 03:08:29.529223 17585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:37481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:08:29.547101 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35035
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:29.874017 17785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:29.874262 17785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:29.874426 17785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:29.881852 17785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:29.881947 17785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:29.882054 17785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:08:29.891579 17785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:29.892689 17785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:29.894142 17785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:29.907083 17790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:29.907272 17793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:29.907294 17791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:29.908205 17785 server_base.cc:1034] running on GCE node
I20260807 03:08:29.908766 17785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:29.909775 17785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:29.911005 17785 hybrid_clock.cc:648] HybridClock initialized: now 1786072109910974 us; error 52 us; skew 500 ppm
I20260807 03:08:29.911310 17785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:29 dist-test-slave-tm1g krb5kdc[17539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072109, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:29.914472 17785 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:08:29.915932 17785 webserver.cc:460] Webserver started at http://127.17.31.2:41223/ using document root <none> and password file <none>
I20260807 03:08:29.916551 17785 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:29.916647 17785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:29.916896 17785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:29.919584 17785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "09a7200376744c9284c2739f7866d2ad"
format_stamp: "Formatted at 2026-08-07 03:08:29 on dist-test-slave-tm1g"
server_key: "269a9486e1c50d705279c9706e0a3b38"
server_key_iv: "4a6b774a0905e778f5b79635e60918a9"
server_key_version: "encryptionkey@0"
I20260807 03:08:29.920275 17785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "09a7200376744c9284c2739f7866d2ad"
format_stamp: "Formatted at 2026-08-07 03:08:29 on dist-test-slave-tm1g"
server_key: "269a9486e1c50d705279c9706e0a3b38"
server_key_iv: "4a6b774a0905e778f5b79635e60918a9"
server_key_version: "encryptionkey@0"
I20260807 03:08:29.924990 17785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260807 03:08:29.928285 17800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:29.929498 17785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:08:29.929649 17785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "09a7200376744c9284c2739f7866d2ad"
format_stamp: "Formatted at 2026-08-07 03:08:29 on dist-test-slave-tm1g"
server_key: "269a9486e1c50d705279c9706e0a3b38"
server_key_iv: "4a6b774a0905e778f5b79635e60918a9"
server_key_version: "encryptionkey@0"
I20260807 03:08:29.929847 17785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:29.942062 17785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:29.945995 17785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:29.946264 17785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:29.947208 17785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:08:29.948398 17785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:29.948468 17785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:29.948632 17785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:29.948684 17785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:29.983749 17785 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:45737
I20260807 03:08:29.983794 17915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:45737 every 8 connection(s)
I20260807 03:08:29.985077 17785 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:08:29.991859 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 17785
I20260807 03:08:29.992077 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:08:29.992511 17532 external_mini_cluster.cc:1428] Setting key 0cb0beaccbef275a7853e35a44201112
Aug 07 03:08:29 dist-test-slave-tm1g krb5kdc[17539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072109, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:30.007076 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:29.987693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:50895 (local address 127.17.31.62:35035)
0807 03:08:29.987864 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:29.987869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:29.989142 (+ 1273us) server_negotiation.cc:407] Connection header received
0807 03:08:29.991011 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:29.991016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:29.991106 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:29.991228 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:29.992736 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:29.993763 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:29.995506 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:29.995926 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:29.996070 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:30.000300 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:30.000305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:30.000309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:30.000338 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:30.003629 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:30.004521 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:30.004530 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:30.004536 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:30.004692 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:30.005094 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:30.005102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:30.005108 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:30.005492 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:30.005641 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:30.006371 (+ 730us) server_negotiation.cc:299] Negotiation successful
0807 03:08:30.006635 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 03:08:30.007663 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:29.988139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35035 (local address 127.17.31.2:50895)
0807 03:08:29.988840 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:29.988883 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:29.990743 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:29.991447 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:29.991455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:29.991929 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:29.992554 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:29.992570 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:29.993935 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:29.993942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:29.995350 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:29.995362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:29.996154 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:29.996162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:29.996384 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:29.997193 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:29.997231 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:30.000095 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:30.003781 (+ 3686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:30.003787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:30.003798 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:30.004361 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:30.004798 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:30.004801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:30.004804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:30.004975 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:30.005617 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:30.005622 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:30.005909 (+ 287us) client_negotiation.cc:769] Sending connection context
0807 03:08:30.006165 (+ 256us) client_negotiation.cc:240] Negotiation successful
0807 03:08:30.006603 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":289,"threads_started":1}
I20260807 03:08:30.009018 17916 heartbeater.cc:347] Connected to a master server at 127.17.31.62:35035
I20260807 03:08:30.009315 17916 heartbeater.cc:464] Registering TS with master...
I20260807 03:08:30.010035 17916 heartbeater.cc:511] Master 127.17.31.62:35035 requested a full tablet report, sending...
I20260807 03:08:30.011610 17585 ts_manager.cc:194] Registered new tserver with Master: 09a7200376744c9284c2739f7866d2ad (127.17.31.2:45737)
I20260807 03:08:30.012441 17585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:50895
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:08:30.036732 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35035
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:42755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:30.358412 17922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:30.358670 17922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:30.358839 17922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:30.365828 17922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:30.365955 17922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:30.366151 17922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:08:30.375365 17922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:42755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:30.376543 17922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:30.377815 17922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:30.391081 17927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:30.391130 17928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:30.391201 17931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:30.391844 17922 server_base.cc:1034] running on GCE node
I20260807 03:08:30.392441 17922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:30.393388 17922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:30.394629 17922 hybrid_clock.cc:648] HybridClock initialized: now 1786072110394575 us; error 33 us; skew 500 ppm
I20260807 03:08:30.394936 17922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:30 dist-test-slave-tm1g krb5kdc[17539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072110, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:30.397929 17922 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:08:30.399295 17922 webserver.cc:460] Webserver started at http://127.17.31.3:40447/ using document root <none> and password file <none>
I20260807 03:08:30.399960 17922 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:30.400116 17922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:30.400418 17922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:30.403370 17922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "e1db6afb04eb490f8777ca0f07aae5b0"
format_stamp: "Formatted at 2026-08-07 03:08:30 on dist-test-slave-tm1g"
server_key: "dc1d5d94c38fc3cc59a5aecedfa91f57"
server_key_iv: "1789a9d4fa8e559a338262e43fcd8cf2"
server_key_version: "encryptionkey@0"
I20260807 03:08:30.404142 17922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "e1db6afb04eb490f8777ca0f07aae5b0"
format_stamp: "Formatted at 2026-08-07 03:08:30 on dist-test-slave-tm1g"
server_key: "dc1d5d94c38fc3cc59a5aecedfa91f57"
server_key_iv: "1789a9d4fa8e559a338262e43fcd8cf2"
server_key_version: "encryptionkey@0"
I20260807 03:08:30.409292 17922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260807 03:08:30.412593 17938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:30.413832 17922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 03:08:30.414012 17922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "e1db6afb04eb490f8777ca0f07aae5b0"
format_stamp: "Formatted at 2026-08-07 03:08:30 on dist-test-slave-tm1g"
server_key: "dc1d5d94c38fc3cc59a5aecedfa91f57"
server_key_iv: "1789a9d4fa8e559a338262e43fcd8cf2"
server_key_version: "encryptionkey@0"
I20260807 03:08:30.414253 17922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:30.445814 17922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:30.449666 17922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:30.449954 17922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:30.450985 17922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:08:30.452340 17922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:30.452443 17922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:30.452606 17922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:30.452687 17922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:30.487087 17922 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:37265
I20260807 03:08:30.487171 18053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:37265 every 8 connection(s)
I20260807 03:08:30.488346 17922 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:08:30.492622 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 17922
I20260807 03:08:30.492849 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:08:30.493450 17532 external_mini_cluster.cc:1428] Setting key f63777bee9a5e9e6738f84e4f583357d
Aug 07 03:08:30 dist-test-slave-tm1g krb5kdc[17539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072110, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:30.511992 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:30.490823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:57399 (local address 127.17.31.62:35035)
0807 03:08:30.491041 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:30.491048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:30.492259 (+ 1211us) server_negotiation.cc:407] Connection header received
0807 03:08:30.494833 (+ 2574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:30.494838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:30.494947 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:30.495060 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:30.497095 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:30.498024 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:30.499753 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:30.500190 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:30.500331 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:30.506525 (+ 6194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:30.506530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:30.506533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:30.506563 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:30.509064 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:30.509934 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:30.509938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:30.509940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:30.510084 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:30.510608 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:30.510613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:30.510616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:30.510897 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:30.511010 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:30.511603 (+ 593us) server_negotiation.cc:299] Negotiation successful
0807 03:08:30.511757 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 03:08:30.512962 18056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:30.491369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35035 (local address 127.17.31.3:57399)
0807 03:08:30.492076 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:30.492112 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:30.494538 (+ 2426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:30.495524 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:30.495541 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:30.496326 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:30.496919 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:30.496935 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:30.498188 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:30.498193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:30.499611 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:30.499623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:30.500422 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:30.500430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:30.500674 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:30.504226 (+ 3552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:30.504248 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:30.506288 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:30.509183 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:30.509193 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:30.509214 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:30.509793 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:30.510201 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:30.510206 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:30.510210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:30.510474 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:30.511008 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:30.511013 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:30.511288 (+ 275us) client_negotiation.cc:769] Sending connection context
0807 03:08:30.511499 (+ 211us) client_negotiation.cc:240] Negotiation successful
0807 03:08:30.511902 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":297,"threads_started":1}
I20260807 03:08:30.514230 18054 heartbeater.cc:347] Connected to a master server at 127.17.31.62:35035
I20260807 03:08:30.514572 18054 heartbeater.cc:464] Registering TS with master...
I20260807 03:08:30.515208 18054 heartbeater.cc:511] Master 127.17.31.62:35035 requested a full tablet report, sending...
I20260807 03:08:30.516551 17585 ts_manager.cc:194] Registered new tserver with Master: e1db6afb04eb490f8777ca0f07aae5b0 (127.17.31.3:37265)
I20260807 03:08:30.517426 17585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:57399
I20260807 03:08:30.522624 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:08:30.532511 17778 heartbeater.cc:503] Master 127.17.31.62:35035 was elected leader, sending a full tablet report...
Aug 07 03:08:30 dist-test-slave-tm1g krb5kdc[17539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072110, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 03:08:30 dist-test-slave-tm1g krb5kdc[17539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072110, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:08:30.570837 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:30.556042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.17.31.1:42853)
0807 03:08:30.556705 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:30.556716 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:30.556758 (+ 42us) server_negotiation.cc:407] Connection header received
0807 03:08:30.556949 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:30.556956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:30.557168 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:30.557337 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:30.558824 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:30.559834 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:30.561087 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:30.561644 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:30.561779 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:30.565564 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:30.565574 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:30.565586 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:30.565616 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:30.568239 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:30.568834 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:30.568842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:30.568844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:30.568962 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:30.569322 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:30.569328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:30.569332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:30.569655 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:30.569940 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:30.570221 (+ 281us) server_negotiation.cc:299] Negotiation successful
0807 03:08:30.570499 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":338,"threads_started":1}
W20260807 03:08:30.572117 17692 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42278
Aug 07 03:08:30 dist-test-slave-tm1g krb5kdc[17539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 03:08:30 dist-test-slave-tm1g krb5kdc[17539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:08:30.616320 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:30.602566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.17.31.1:42853)
0807 03:08:30.602761 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:30.602770 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:30.603097 (+ 327us) server_negotiation.cc:407] Connection header received
0807 03:08:30.603298 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:30.603304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:30.603407 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:30.603528 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:30.604856 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:30.605758 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:30.607079 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:30.607498 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:30.607632 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:30.611319 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:30.611324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:30.611327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:30.611359 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:30.613726 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:30.614375 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:30.614387 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:30.614391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:30.614516 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:30.614972 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:30.614977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:30.614982 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:30.615232 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:30.615488 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:30.615786 (+ 298us) server_negotiation.cc:299] Negotiation successful
0807 03:08:30.616004 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 03:08:30.617980 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 17647
I20260807 03:08:30.629665 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:30.619047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42300 (local address 127.17.31.1:42853)
0807 03:08:30.619191 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:30.619195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:30.619515 (+ 320us) server_negotiation.cc:407] Connection header received
0807 03:08:30.619617 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:30.619620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:30.619669 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:30.619740 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:30.620816 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:30.621445 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:30.622676 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:30.623096 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:30.623186 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:30.625215 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:30.625220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:30.625223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:30.625249 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:30.627617 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:30.628210 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:30.628213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:30.628216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:30.628331 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:30.628622 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:30.628625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:30.628627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:30.628845 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:30.629014 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:30.629271 (+ 257us) server_negotiation.cc:299] Negotiation successful
0807 03:08:30.629405 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 03:08:30.630467 17773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:08:30.716673 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 17647
I20260807 03:08:30.734766 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 17785
Aug 07 03:08:30 dist-test-slave-tm1g krb5kdc[17539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:08:30.749876 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:30.735671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.17.31.2:45737)
0807 03:08:30.736174 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:30.736181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:30.736206 (+ 25us) server_negotiation.cc:407] Connection header received
0807 03:08:30.736266 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:30.736272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:30.736487 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:30.736611 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:30.737833 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:30.738928 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:30.740231 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:30.740769 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:30.740863 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:30.744750 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:30.744759 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:30.744771 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:30.744800 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:30.747439 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:30.748007 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:30.748013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:30.748015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:30.748134 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:30.748471 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:30.748475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:30.748477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:30.748805 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:30.749046 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:30.749291 (+ 245us) server_negotiation.cc:299] Negotiation successful
0807 03:08:30.749476 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":256,"threads_started":1}
I20260807 03:08:30.751181 17911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:08:30.833029 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 17785
I20260807 03:08:30.851516 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 17922
Aug 07 03:08:30 dist-test-slave-tm1g krb5kdc[17539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:08:30.869021 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:30.852630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46326 (local address 127.17.31.3:37265)
0807 03:08:30.853382 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:30.853391 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:30.853427 (+ 36us) server_negotiation.cc:407] Connection header received
0807 03:08:30.853500 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:30.853509 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:30.853777 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:30.853939 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:30.855194 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:30.856824 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:30.858061 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:30.858976 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:30.859107 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:30.864006 (+ 4899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:30.864016 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:30.864030 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:30.864062 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:30.866808 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:30.867381 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:30.867388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:30.867391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:30.867518 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:30.867812 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:30.867814 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:30.867816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:30.868134 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:30.868353 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:30.868559 (+ 206us) server_negotiation.cc:299] Negotiation successful
0807 03:08:30.868747 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":406,"threads_started":1}
I20260807 03:08:30.870393 18049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:08:30.953083 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 17922
I20260807 03:08:30.970762 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 17552
I20260807 03:08:30.971992 17616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:08:31.073402 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 17552
2026-08-07T03:08:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4017 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:08:31 dist-test-slave-tm1g krb5kdc[18076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:08:31 dist-test-slave-tm1g krb5kdc[18076](info): set up 2 sockets
Aug 07 03:08:31 dist-test-slave-tm1g krb5kdc[18076](info): commencing operation
krb5kdc: starting...
W20260807 03:08:32.175609 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:08:32 dist-test-slave-tm1g krb5kdc[18076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:08:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:08:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:08:32.346273 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:43211
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:43211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:32.664224 18089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:32.664505 18089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:32.664595 18089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:32.671836 18089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:08:32.671933 18089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:32.672025 18089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:32.672125 18089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:08:32.672210 18089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:08:32.681805 18089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:43211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:43211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:32.683086 18089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:32.684371 18089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:32.695856 18095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:32.695812 18096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:32.695773 18098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:32.696780 18089 server_base.cc:1034] running on GCE node
I20260807 03:08:32.697571 18089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:32.698967 18089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:32.700196 18089 hybrid_clock.cc:648] HybridClock initialized: now 1786072112700165 us; error 44 us; skew 500 ppm
I20260807 03:08:32.700543 18089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:32 dist-test-slave-tm1g krb5kdc[18076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072112, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:32.703507 18089 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:08:32.704908 18089 webserver.cc:460] Webserver started at http://127.17.31.62:33103/ using document root <none> and password file <none>
I20260807 03:08:32.705574 18089 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:32.705667 18089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:32.705919 18089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:32.708747 18089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "a54f27beda07408a88301a300e86bfb4"
format_stamp: "Formatted at 2026-08-07 03:08:32 on dist-test-slave-tm1g"
server_key: "db8253bf983f0366989bd8793b133821"
server_key_iv: "997084c7d55d79ab27041eae6a2ac071"
server_key_version: "encryptionkey@0"
I20260807 03:08:32.709424 18089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "a54f27beda07408a88301a300e86bfb4"
format_stamp: "Formatted at 2026-08-07 03:08:32 on dist-test-slave-tm1g"
server_key: "db8253bf983f0366989bd8793b133821"
server_key_iv: "997084c7d55d79ab27041eae6a2ac071"
server_key_version: "encryptionkey@0"
I20260807 03:08:32.714166 18089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260807 03:08:32.717512 18106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:32.718827 18089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:08:32.719012 18089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "a54f27beda07408a88301a300e86bfb4"
format_stamp: "Formatted at 2026-08-07 03:08:32 on dist-test-slave-tm1g"
server_key: "db8253bf983f0366989bd8793b133821"
server_key_iv: "997084c7d55d79ab27041eae6a2ac071"
server_key_version: "encryptionkey@0"
I20260807 03:08:32.719187 18089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:32.748945 18089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:32.752727 18089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:32.753010 18089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:32.770944 18089 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:43211
I20260807 03:08:32.771013 18159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:43211 every 8 connection(s)
I20260807 03:08:32.772289 18089 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:08:32.775810 18160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:32.781303 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 18089
I20260807 03:08:32.781545 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:08:32.782197 17532 external_mini_cluster.cc:1428] Setting key f1a87995b215294cb2b1f2531139120b
I20260807 03:08:32.786177 18160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4: Bootstrap starting.
I20260807 03:08:32.789842 18160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:32.791007 18160 log.cc:824] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4: Log is configured to *not* fsync() on all Append() calls
I20260807 03:08:32.793922 18160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4: No bootstrap required, opened a new log
Aug 07 03:08:32 dist-test-slave-tm1g krb5kdc[18076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:32.801918 18160 raft_consensus.cc:348] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54f27beda07408a88301a300e86bfb4" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 43211 } }
I20260807 03:08:32.802281 18160 raft_consensus.cc:374] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:32.802464 18160 raft_consensus.cc:729] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a54f27beda07408a88301a300e86bfb4, State: Initialized, Role: FOLLOWER
I20260807 03:08:32.803153 18160 consensus_queue.cc:260] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54f27beda07408a88301a300e86bfb4" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 43211 } }
I20260807 03:08:32.803458 18160 raft_consensus.cc:388] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:08:32.803620 18160 raft_consensus.cc:482] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:08:32.803817 18160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:32.805056 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:32.784177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43810 (local address 127.17.31.62:43211)
0807 03:08:32.784930 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:32.784940 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:32.784979 (+ 39us) server_negotiation.cc:407] Connection header received
0807 03:08:32.786704 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:32.786723 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:32.787075 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:32.787366 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:32.789290 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:32.790662 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:32.792237 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:32.792849 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:32.792977 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:32.797606 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:32.797616 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:32.797628 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:32.797663 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:32.800746 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:32.801431 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:32.801438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:32.801444 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:32.801602 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:32.801991 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:32.801997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:32.802000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:32.802705 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:32.802999 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:32.803580 (+ 581us) server_negotiation.cc:299] Negotiation successful
0807 03:08:32.804021 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":368,"threads_started":1}
I20260807 03:08:32.806245 18160 raft_consensus.cc:504] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54f27beda07408a88301a300e86bfb4" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 43211 } }
I20260807 03:08:32.806798 18160 leader_election.cc:302] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a54f27beda07408a88301a300e86bfb4; no voters:
I20260807 03:08:32.807480 18160 leader_election.cc:288] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:08:32.807593 18165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:08:32.810009 18160 sys_catalog.cc:564] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:08:32.810374 18165 raft_consensus.cc:686] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [term 1 LEADER]: Becoming Leader. State: Replica: a54f27beda07408a88301a300e86bfb4, State: Running, Role: LEADER
I20260807 03:08:32.810945 18165 consensus_queue.cc:237] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54f27beda07408a88301a300e86bfb4" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 43211 } }
I20260807 03:08:32.814457 18167 sys_catalog.cc:455] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a54f27beda07408a88301a300e86bfb4. Latest consensus state: current_term: 1 leader_uuid: "a54f27beda07408a88301a300e86bfb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54f27beda07408a88301a300e86bfb4" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 43211 } } }
I20260807 03:08:32.814885 18167 sys_catalog.cc:458] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [sys.catalog]: This master's current role is: LEADER
I20260807 03:08:32.815325 18166 sys_catalog.cc:455] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a54f27beda07408a88301a300e86bfb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a54f27beda07408a88301a300e86bfb4" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 43211 } } }
I20260807 03:08:32.815608 18166 sys_catalog.cc:458] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4 [sys.catalog]: This master's current role is: LEADER
I20260807 03:08:32.816797 18178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:08:32.820910 18178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:08:32.829500 18178 catalog_manager.cc:1347] Generated new cluster ID: bbb17441470042a19deeb16e44a6bb6e
I20260807 03:08:32.829612 18178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:08:32.838122 18178 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:08:32.839432 18178 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:08:32.854437 18178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a54f27beda07408a88301a300e86bfb4: Generated new TSK 0
I20260807 03:08:32.855290 18178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:08:32.899444 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:43211
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:33.217367 18187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:33.217619 18187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:33.217712 18187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:33.224738 18187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:33.224831 18187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:33.224948 18187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:08:33.234249 18187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:43211
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:33.235435 18187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:33.236656 18187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:33.249191 18195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:33.249316 18192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:33.249285 18193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:33.250025 18187 server_base.cc:1034] running on GCE node
I20260807 03:08:33.250631 18187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:33.251441 18187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:33.252629 18187 hybrid_clock.cc:648] HybridClock initialized: now 1786072113252590 us; error 37 us; skew 500 ppm
I20260807 03:08:33.252928 18187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:33 dist-test-slave-tm1g krb5kdc[18076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072113, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:33.255887 18187 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:08:33.257292 18187 webserver.cc:460] Webserver started at http://127.17.31.1:46509/ using document root <none> and password file <none>
I20260807 03:08:33.257939 18187 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:33.258083 18187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:33.258458 18187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:33.261196 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "71a9cd7cdc4e4e9c806a8256bede361e"
format_stamp: "Formatted at 2026-08-07 03:08:33 on dist-test-slave-tm1g"
server_key: "8554c04494f1e8464b1f4d2ea6f331a0"
server_key_iv: "584943a15153734a6a73e9d9f3fc5881"
server_key_version: "encryptionkey@0"
I20260807 03:08:33.261976 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "71a9cd7cdc4e4e9c806a8256bede361e"
format_stamp: "Formatted at 2026-08-07 03:08:33 on dist-test-slave-tm1g"
server_key: "8554c04494f1e8464b1f4d2ea6f331a0"
server_key_iv: "584943a15153734a6a73e9d9f3fc5881"
server_key_version: "encryptionkey@0"
I20260807 03:08:33.266829 18187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 03:08:33.269975 18202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:33.271238 18187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:08:33.271409 18187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "71a9cd7cdc4e4e9c806a8256bede361e"
format_stamp: "Formatted at 2026-08-07 03:08:33 on dist-test-slave-tm1g"
server_key: "8554c04494f1e8464b1f4d2ea6f331a0"
server_key_iv: "584943a15153734a6a73e9d9f3fc5881"
server_key_version: "encryptionkey@0"
I20260807 03:08:33.271591 18187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:33.282972 18187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:33.286631 18187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:33.286913 18187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:33.287746 18187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:08:33.288954 18187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:33.289031 18187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:33.289186 18187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:33.289247 18187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:33.321255 18187 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:46705
I20260807 03:08:33.321302 18321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:46705 every 8 connection(s)
I20260807 03:08:33.322769 18187 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:08:33.325408 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 18187
I20260807 03:08:33.325636 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:08:33.326527 17532 external_mini_cluster.cc:1428] Setting key af7eea6ebedbc26c613567048cd91b8a
Aug 07 03:08:33 dist-test-slave-tm1g krb5kdc[18076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072113, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:33.342897 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:33.325214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:58249 (local address 127.17.31.62:43211)
0807 03:08:33.325659 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:33.325664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:33.326926 (+ 1262us) server_negotiation.cc:407] Connection header received
0807 03:08:33.328946 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:33.328952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:33.329039 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:33.329154 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:33.330637 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:33.331355 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:33.332924 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:33.333334 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:33.333433 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:33.337442 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:33.337447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:33.337451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:33.337481 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:33.340012 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:33.340693 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:33.340702 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:33.340704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:33.340816 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:33.341234 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:33.341238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:33.341242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:33.341522 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:33.341675 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:33.342320 (+ 645us) server_negotiation.cc:299] Negotiation successful
0807 03:08:33.342547 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":237,"threads_started":1}
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
I20260807 03:08:33.343748 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:33.325921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:43211 (local address 127.17.31.1:58249)
0807 03:08:33.326777 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:33.326813 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:33.328689 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:33.329373 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:33.329381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:33.329862 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:33.330474 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:33.330492 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:33.331509 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:33.331514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:33.332769 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:33.332794 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:33.333575 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:33.333584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:33.333920 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:33.334654 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:33.334687 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:33.337233 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:33.340146 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:33.340154 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:33.340168 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:33.340569 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:33.340922 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:33.340927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:33.340931 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:33.341114 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:33.341644 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:33.341652 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:33.341944 (+ 292us) client_negotiation.cc:769] Sending connection context
0807 03:08:33.342192 (+ 248us) client_negotiation.cc:240] Negotiation successful
0807 03:08:33.342679 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":385,"threads_started":1}
I20260807 03:08:33.345358 18322 heartbeater.cc:347] Connected to a master server at 127.17.31.62:43211
I20260807 03:08:33.345683 18322 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
I20260807 03:08:33.346509 18322 heartbeater.cc:511] Master 127.17.31.62:43211 requested a full tablet report, sending...
I20260807 03:08:33.348631 18125 ts_manager.cc:194] Registered new tserver with Master: 71a9cd7cdc4e4e9c806a8256bede361e (127.17.31.1:46705)
I20260807 03:08:33.350090 18125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:58249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:08:33.367069 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:43211
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:33.675992 18330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:33.676555 18330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:33.676738 18330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:33.683842 18330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:33.683938 18330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:33.684234 18330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:08:33.693598 18330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:43211
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:33.694808 18330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:33.696085 18330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:33.709105 18335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:33.709136 18336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:33.709168 18338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:33.710186 18330 server_base.cc:1034] running on GCE node
I20260807 03:08:33.710857 18330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:33.711784 18330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:33.712999 18330 hybrid_clock.cc:648] HybridClock initialized: now 1786072113712950 us; error 53 us; skew 500 ppm
I20260807 03:08:33.713361 18330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:33 dist-test-slave-tm1g krb5kdc[18076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072113, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:33.716529 18330 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:08:33.717826 18330 webserver.cc:460] Webserver started at http://127.17.31.2:45663/ using document root <none> and password file <none>
I20260807 03:08:33.718575 18330 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:33.718715 18330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:33.719028 18330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:33.721799 18330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "f910cbe657fa4aadb5aeafdd547de0b5"
format_stamp: "Formatted at 2026-08-07 03:08:33 on dist-test-slave-tm1g"
server_key: "feb3243c7d21d0d977c5c1c7d31486c3"
server_key_iv: "21729b196186d2aa1254186e265fcd12"
server_key_version: "encryptionkey@0"
I20260807 03:08:33.722729 18330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "f910cbe657fa4aadb5aeafdd547de0b5"
format_stamp: "Formatted at 2026-08-07 03:08:33 on dist-test-slave-tm1g"
server_key: "feb3243c7d21d0d977c5c1c7d31486c3"
server_key_iv: "21729b196186d2aa1254186e265fcd12"
server_key_version: "encryptionkey@0"
I20260807 03:08:33.727604 18330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260807 03:08:33.730824 18345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:33.732028 18330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:08:33.732196 18330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "f910cbe657fa4aadb5aeafdd547de0b5"
format_stamp: "Formatted at 2026-08-07 03:08:33 on dist-test-slave-tm1g"
server_key: "feb3243c7d21d0d977c5c1c7d31486c3"
server_key_iv: "21729b196186d2aa1254186e265fcd12"
server_key_version: "encryptionkey@0"
I20260807 03:08:33.732383 18330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:33.756608 18330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:33.760793 18330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:33.761126 18330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:33.762151 18330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:08:33.763417 18330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:33.763495 18330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:33.763592 18330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:33.763636 18330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:33.801169 18330 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:38483
I20260807 03:08:33.801291 18462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:38483 every 8 connection(s)
I20260807 03:08:33.802430 18330 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:08:33.811579 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 18330
I20260807 03:08:33.811774 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:08:33.812328 17532 external_mini_cluster.cc:1428] Setting key d4990e16570bfaf35defebedf93eace9
Aug 07 03:08:33 dist-test-slave-tm1g krb5kdc[18076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072113, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:33.823766 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:33.804805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:60037 (local address 127.17.31.62:43211)
0807 03:08:33.804980 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:33.804985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:33.806130 (+ 1145us) server_negotiation.cc:407] Connection header received
0807 03:08:33.807982 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:33.807989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:33.808097 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:33.808210 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:33.809614 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:33.810519 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:33.812584 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:33.813029 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:33.813179 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:33.817143 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:33.817152 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:33.817159 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:33.817204 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:33.820661 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:33.821349 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:33.821357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:33.821360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:33.821505 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:33.821999 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:33.822003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:33.822005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:33.822253 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:33.822480 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:33.823216 (+ 736us) server_negotiation.cc:299] Negotiation successful
0807 03:08:33.823433 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 03:08:33.824767 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:33.805277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:43211 (local address 127.17.31.2:60037)
0807 03:08:33.805969 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:33.806013 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:33.807726 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:33.808426 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:33.808434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:33.808921 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:33.809449 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:33.809465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:33.810655 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:33.810663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:33.812421 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:33.812439 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:33.813273 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:33.813281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:33.813540 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:33.814286 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:33.814348 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:33.816907 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:33.820798 (+ 3891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:33.820806 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:33.820820 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:33.821221 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:33.821618 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:33.821626 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:33.821632 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:33.821859 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:33.822452 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:33.822460 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:33.822769 (+ 309us) client_negotiation.cc:769] Sending connection context
0807 03:08:33.823108 (+ 339us) client_negotiation.cc:240] Negotiation successful
0807 03:08:33.823644 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":278,"threads_started":1}
I20260807 03:08:33.826217 18463 heartbeater.cc:347] Connected to a master server at 127.17.31.62:43211
I20260807 03:08:33.826553 18463 heartbeater.cc:464] Registering TS with master...
I20260807 03:08:33.827227 18463 heartbeater.cc:511] Master 127.17.31.62:43211 requested a full tablet report, sending...
I20260807 03:08:33.828685 18125 ts_manager.cc:194] Registered new tserver with Master: f910cbe657fa4aadb5aeafdd547de0b5 (127.17.31.2:38483)
I20260807 03:08:33.829452 18125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:60037
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:08:33.853282 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:43211
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:34.165787 18468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:34.166074 18468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:34.166249 18468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:34.173091 18468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:34.173223 18468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:34.173414 18468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:08:34.182664 18468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:43211
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:34.183854 18468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:34.185117 18468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:34.197499 18474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:34.197547 18476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:34.197563 18473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:34.198539 18468 server_base.cc:1034] running on GCE node
I20260807 03:08:34.199127 18468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:34.200019 18468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:34.201234 18468 hybrid_clock.cc:648] HybridClock initialized: now 1786072114201198 us; error 54 us; skew 500 ppm
I20260807 03:08:34.201524 18468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:34 dist-test-slave-tm1g krb5kdc[18076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072114, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:34.204454 18468 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:08:34.205760 18468 webserver.cc:460] Webserver started at http://127.17.31.3:43247/ using document root <none> and password file <none>
I20260807 03:08:34.206472 18468 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:34.206624 18468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:34.206887 18468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:34.209524 18468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "b7ea26ba7b464657a56109bc6e6280a0"
format_stamp: "Formatted at 2026-08-07 03:08:34 on dist-test-slave-tm1g"
server_key: "9ccf74b52c03aba333edb71934447a16"
server_key_iv: "3ceee41d9491df1bd6389a5dfc76e730"
server_key_version: "encryptionkey@0"
I20260807 03:08:34.210233 18468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "b7ea26ba7b464657a56109bc6e6280a0"
format_stamp: "Formatted at 2026-08-07 03:08:34 on dist-test-slave-tm1g"
server_key: "9ccf74b52c03aba333edb71934447a16"
server_key_iv: "3ceee41d9491df1bd6389a5dfc76e730"
server_key_version: "encryptionkey@0"
I20260807 03:08:34.215126 18468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 03:08:34.218235 18483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:34.219519 18468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:08:34.219686 18468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "b7ea26ba7b464657a56109bc6e6280a0"
format_stamp: "Formatted at 2026-08-07 03:08:34 on dist-test-slave-tm1g"
server_key: "9ccf74b52c03aba333edb71934447a16"
server_key_iv: "3ceee41d9491df1bd6389a5dfc76e730"
server_key_version: "encryptionkey@0"
I20260807 03:08:34.219893 18468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:34.245589 18468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:34.249389 18468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:34.249662 18468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:34.250696 18468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:08:34.251947 18468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:34.252018 18468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:34.252192 18468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:34.252240 18468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:34.284525 18468 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:44117
I20260807 03:08:34.284610 18601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:44117 every 8 connection(s)
I20260807 03:08:34.285779 18468 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:08:34.287933 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 18468
I20260807 03:08:34.288174 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:08:34.288667 17532 external_mini_cluster.cc:1428] Setting key b6e55e9f0629818919c79d331e6e503c
Aug 07 03:08:34 dist-test-slave-tm1g krb5kdc[18076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072114, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:34.304967 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.288301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:39571 (local address 127.17.31.62:43211)
0807 03:08:34.288493 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:34.288499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:34.289737 (+ 1238us) server_negotiation.cc:407] Connection header received
0807 03:08:34.291786 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:34.291790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:34.291852 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:34.291927 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:34.293404 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.294123 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.295665 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.296058 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.296146 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.299379 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:34.299383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:34.299387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:34.299415 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:34.301772 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:34.302563 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:34.302568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:34.302571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:34.302731 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:34.303217 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:34.303225 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:34.303229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:34.303580 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:34.303720 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:34.304417 (+ 697us) server_negotiation.cc:299] Negotiation successful
0807 03:08:34.304641 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 03:08:34.305821 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.288856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:43211 (local address 127.17.31.3:39571)
0807 03:08:34.289586 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:34.289622 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:34.291511 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:34.292119 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:34.292127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:34.292612 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:34.293248 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:34.293271 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.294265 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.294269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:34.295543 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:34.295560 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.296230 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.296234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:34.296413 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.297082 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:34.297103 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:34.299196 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:34.301911 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:34.301918 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:34.301932 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:34.302457 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:34.302849 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:34.302853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:34.302856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:34.303097 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:34.303706 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:34.303713 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:34.304077 (+ 364us) client_negotiation.cc:769] Sending connection context
0807 03:08:34.304318 (+ 241us) client_negotiation.cc:240] Negotiation successful
0807 03:08:34.304780 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":327,"threads_started":1}
I20260807 03:08:34.307085 18602 heartbeater.cc:347] Connected to a master server at 127.17.31.62:43211
I20260807 03:08:34.307371 18602 heartbeater.cc:464] Registering TS with master...
I20260807 03:08:34.308053 18602 heartbeater.cc:511] Master 127.17.31.62:43211 requested a full tablet report, sending...
I20260807 03:08:34.309309 18125 ts_manager.cc:194] Registered new tserver with Master: b7ea26ba7b464657a56109bc6e6280a0 (127.17.31.3:44117)
I20260807 03:08:34.310036 18125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:39571
I20260807 03:08:34.315858 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:08:34.335695 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.323589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43826 (local address 127.17.31.62:43211)
0807 03:08:34.323756 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:34.323761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:34.324310 (+ 549us) server_negotiation.cc:407] Connection header received
0807 03:08:34.324465 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:34.324469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:34.324530 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:34.324603 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:34.326557 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.327241 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.328523 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.328902 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.328987 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.331229 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:34.331234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:34.331237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:34.331263 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:34.333468 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:34.334043 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:34.334046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:34.334048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:34.334156 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:34.334523 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:34.334526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:34.334528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:34.334805 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:34.334979 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:34.335238 (+ 259us) server_negotiation.cc:299] Negotiation successful
0807 03:08:34.335433 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:08:34.343569 18125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:08:34.351816 18125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:08:34.352785 18322 heartbeater.cc:503] Master 127.17.31.62:43211 was elected leader, sending a full tablet report...
I20260807 03:08:34.388510 18615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.367351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.17.31.3:44117)
0807 03:08:34.380122 (+ 12771us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:34.380136 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:34.380176 (+ 40us) server_negotiation.cc:407] Connection header received
0807 03:08:34.380297 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:34.380309 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:34.380619 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:34.380858 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:34.381727 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.383491 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.385788 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.387296 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.387433 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.387656 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:34.387993 (+ 337us) server_negotiation.cc:299] Negotiation successful
0807 03:08:34.388176 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12546,"thread_start_us":13770,"threads_started":1}
I20260807 03:08:34.389302 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.367070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:44117 (local address 127.0.0.1:44722)
0807 03:08:34.373644 (+ 6574us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:34.373679 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:34.373889 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:34.380986 (+ 7097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:34.380997 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:34.381033 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:34.381481 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:34.381497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.383649 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.383653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:34.385618 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:34.385632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.387513 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.387520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:34.387727 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.387774 (+ 47us) client_negotiation.cc:769] Sending connection context
0807 03:08:34.387946 (+ 172us) client_negotiation.cc:240] Negotiation successful
0807 03:08:34.388117 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6332,"mutex_wait_us":1,"thread_start_us":6207,"threads_started":1}
I20260807 03:08:34.389746 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.367546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:46705 (local address 127.0.0.1:38044)
0807 03:08:34.380518 (+ 12972us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:34.380539 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:34.380664 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:34.382697 (+ 2033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:34.382705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:34.382734 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:34.383234 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:34.383249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.385246 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.385252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:34.386932 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:34.386944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.388940 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.388947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:34.389103 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.389149 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 03:08:34.389230 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 03:08:34.389372 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12813,"thread_start_us":13199,"threads_started":1}
I20260807 03:08:34.390585 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.378726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:38483 (local address 127.0.0.1:55246)
0807 03:08:34.382090 (+ 3364us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:34.382106 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:34.382206 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:34.383617 (+ 1411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:34.383622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:34.383659 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:34.384132 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:34.384147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.385895 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.385900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:34.388002 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:34.388016 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.389839 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.389845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:34.390013 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.390041 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 03:08:34.390146 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 03:08:34.390257 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3235,"thread_start_us":3107,"threads_started":1}
I20260807 03:08:34.390697 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.374363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.17.31.2:38483)
0807 03:08:34.382943 (+ 8580us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:34.382952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:34.382982 (+ 30us) server_negotiation.cc:407] Connection header received
0807 03:08:34.383177 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:34.383187 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:34.383437 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:34.383636 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:34.384310 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.385688 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.388185 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.389641 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.389764 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.389956 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:34.390185 (+ 229us) server_negotiation.cc:299] Negotiation successful
0807 03:08:34.390370 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8158,"thread_start_us":8040,"threads_started":1}
I20260807 03:08:34.391291 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.368223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38044 (local address 127.17.31.1:46705)
0807 03:08:34.381829 (+ 13606us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:34.381839 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:34.381870 (+ 31us) server_negotiation.cc:407] Connection header received
0807 03:08:34.381962 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:34.381968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:34.382196 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:34.382403 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:34.383458 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.385104 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.387075 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.388450 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.388574 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.388795 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:34.390015 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0807 03:08:34.390167 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13429,"thread_start_us":13288,"threads_started":1}
I20260807 03:08:34.394073 18256 tablet_service.cc:1467] Processing CreateTablet for tablet 45c2ccf20dd0492a8d8eb711c3e10aa1 (DEFAULT_TABLE table=test-table [id=8b22756cd90f4a60ad49215a5424b564]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:34.394002 18532 tablet_service.cc:1467] Processing CreateTablet for tablet 45c2ccf20dd0492a8d8eb711c3e10aa1 (DEFAULT_TABLE table=test-table [id=8b22756cd90f4a60ad49215a5424b564]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:34.395105 18395 tablet_service.cc:1467] Processing CreateTablet for tablet 45c2ccf20dd0492a8d8eb711c3e10aa1 (DEFAULT_TABLE table=test-table [id=8b22756cd90f4a60ad49215a5424b564]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:34.395706 18532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45c2ccf20dd0492a8d8eb711c3e10aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:34.396402 18395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45c2ccf20dd0492a8d8eb711c3e10aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:34.397850 18256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45c2ccf20dd0492a8d8eb711c3e10aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:34.408461 18620 tablet_bootstrap.cc:492] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P 71a9cd7cdc4e4e9c806a8256bede361e: Bootstrap starting.
I20260807 03:08:34.410827 18622 tablet_bootstrap.cc:492] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0: Bootstrap starting.
I20260807 03:08:34.411432 18621 tablet_bootstrap.cc:492] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P f910cbe657fa4aadb5aeafdd547de0b5: Bootstrap starting.
I20260807 03:08:34.412894 18620 tablet_bootstrap.cc:654] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P 71a9cd7cdc4e4e9c806a8256bede361e: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:34.414189 18620 log.cc:824] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P 71a9cd7cdc4e4e9c806a8256bede361e: Log is configured to *not* fsync() on all Append() calls
I20260807 03:08:34.414425 18622 tablet_bootstrap.cc:654] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:34.414690 18621 tablet_bootstrap.cc:654] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P f910cbe657fa4aadb5aeafdd547de0b5: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:34.415823 18622 log.cc:824] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0: Log is configured to *not* fsync() on all Append() calls
I20260807 03:08:34.416443 18621 log.cc:824] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P f910cbe657fa4aadb5aeafdd547de0b5: Log is configured to *not* fsync() on all Append() calls
I20260807 03:08:34.418375 18620 tablet_bootstrap.cc:492] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P 71a9cd7cdc4e4e9c806a8256bede361e: No bootstrap required, opened a new log
I20260807 03:08:34.418641 18620 ts_tablet_manager.cc:1397] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P 71a9cd7cdc4e4e9c806a8256bede361e: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260807 03:08:34.419343 18622 tablet_bootstrap.cc:492] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0: No bootstrap required, opened a new log
I20260807 03:08:34.419525 18622 ts_tablet_manager.cc:1397] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 03:08:34.419585 18621 tablet_bootstrap.cc:492] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P f910cbe657fa4aadb5aeafdd547de0b5: No bootstrap required, opened a new log
I20260807 03:08:34.419819 18621 ts_tablet_manager.cc:1397] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P f910cbe657fa4aadb5aeafdd547de0b5: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260807 03:08:34.426725 18622 raft_consensus.cc:348] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ea26ba7b464657a56109bc6e6280a0" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44117 } } peers { permanent_uuid: "71a9cd7cdc4e4e9c806a8256bede361e" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46705 } } peers { permanent_uuid: "f910cbe657fa4aadb5aeafdd547de0b5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 38483 } }
I20260807 03:08:34.427098 18622 raft_consensus.cc:374] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:34.427222 18622 raft_consensus.cc:729] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ea26ba7b464657a56109bc6e6280a0, State: Initialized, Role: FOLLOWER
I20260807 03:08:34.428041 18622 consensus_queue.cc:260] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ea26ba7b464657a56109bc6e6280a0" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44117 } } peers { permanent_uuid: "71a9cd7cdc4e4e9c806a8256bede361e" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46705 } } peers { permanent_uuid: "f910cbe657fa4aadb5aeafdd547de0b5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 38483 } }
I20260807 03:08:34.429425 18602 heartbeater.cc:503] Master 127.17.31.62:43211 was elected leader, sending a full tablet report...
I20260807 03:08:34.429371 18620 raft_consensus.cc:348] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P 71a9cd7cdc4e4e9c806a8256bede361e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ea26ba7b464657a56109bc6e6280a0" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44117 } } peers { permanent_uuid: "71a9cd7cdc4e4e9c806a8256bede361e" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46705 } } peers { permanent_uuid: "f910cbe657fa4aadb5aeafdd547de0b5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 38483 } }
I20260807 03:08:34.429756 18620 raft_consensus.cc:374] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P 71a9cd7cdc4e4e9c806a8256bede361e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:34.429877 18620 raft_consensus.cc:729] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P 71a9cd7cdc4e4e9c806a8256bede361e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a9cd7cdc4e4e9c806a8256bede361e, State: Initialized, Role: FOLLOWER
I20260807 03:08:34.430183 18621 raft_consensus.cc:348] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P f910cbe657fa4aadb5aeafdd547de0b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ea26ba7b464657a56109bc6e6280a0" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44117 } } peers { permanent_uuid: "71a9cd7cdc4e4e9c806a8256bede361e" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46705 } } peers { permanent_uuid: "f910cbe657fa4aadb5aeafdd547de0b5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 38483 } }
I20260807 03:08:34.430647 18621 raft_consensus.cc:374] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P f910cbe657fa4aadb5aeafdd547de0b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:34.430464 18620 consensus_queue.cc:260] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P 71a9cd7cdc4e4e9c806a8256bede361e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ea26ba7b464657a56109bc6e6280a0" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44117 } } peers { permanent_uuid: "71a9cd7cdc4e4e9c806a8256bede361e" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46705 } } peers { permanent_uuid: "f910cbe657fa4aadb5aeafdd547de0b5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 38483 } }
I20260807 03:08:34.430743 18621 raft_consensus.cc:729] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P f910cbe657fa4aadb5aeafdd547de0b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f910cbe657fa4aadb5aeafdd547de0b5, State: Initialized, Role: FOLLOWER
I20260807 03:08:34.431362 18621 consensus_queue.cc:260] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P f910cbe657fa4aadb5aeafdd547de0b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ea26ba7b464657a56109bc6e6280a0" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44117 } } peers { permanent_uuid: "71a9cd7cdc4e4e9c806a8256bede361e" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46705 } } peers { permanent_uuid: "f910cbe657fa4aadb5aeafdd547de0b5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 38483 } }
I20260807 03:08:34.432540 18620 ts_tablet_manager.cc:1428] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P 71a9cd7cdc4e4e9c806a8256bede361e: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260807 03:08:34.433766 18463 heartbeater.cc:503] Master 127.17.31.62:43211 was elected leader, sending a full tablet report...
I20260807 03:08:34.434466 18621 ts_tablet_manager.cc:1428] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P f910cbe657fa4aadb5aeafdd547de0b5: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260807 03:08:34.434684 18622 ts_tablet_manager.cc:1428] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0: Time spent starting tablet: real 0.015s user 0.007s sys 0.004s
W20260807 03:08:34.537416 18603 tablet.cc:2369] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:08:34.554351 18464 tablet.cc:2369] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P f910cbe657fa4aadb5aeafdd547de0b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:08:34.575075 18324 tablet.cc:2369] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P 71a9cd7cdc4e4e9c806a8256bede361e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:08:34.597023 18626 raft_consensus.cc:482] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:08:34.597342 18626 raft_consensus.cc:504] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ea26ba7b464657a56109bc6e6280a0" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44117 } } peers { permanent_uuid: "71a9cd7cdc4e4e9c806a8256bede361e" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46705 } } peers { permanent_uuid: "f910cbe657fa4aadb5aeafdd547de0b5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 38483 } }
I20260807 03:08:34.599620 18626 leader_election.cc:288] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a9cd7cdc4e4e9c806a8256bede361e (127.17.31.1:46705), f910cbe657fa4aadb5aeafdd547de0b5 (127.17.31.2:38483)
I20260807 03:08:34.606838 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.599993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:42123 (local address 127.17.31.1:46705)
0807 03:08:34.600210 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:34.600216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:34.600524 (+ 308us) server_negotiation.cc:407] Connection header received
0807 03:08:34.600683 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:34.600688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:34.600776 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:34.600850 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:34.601302 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.602362 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.604813 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.605973 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.606110 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.606214 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:34.606518 (+ 304us) server_negotiation.cc:299] Negotiation successful
0807 03:08:34.606640 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 03:08:34.606923 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.599953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:46705 (local address 127.17.31.3:42123)
0807 03:08:34.600339 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:34.600356 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:34.600523 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:34.600901 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:34.600908 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:34.600945 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:34.601158 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:34.601173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.602476 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.602480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:34.604648 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:34.604667 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.606131 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.606135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:34.606349 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.606382 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 03:08:34.606455 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 03:08:34.606608 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":58}
I20260807 03:08:34.607438 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.600091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:38483 (local address 127.17.31.3:40357)
0807 03:08:34.600817 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:34.600852 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:34.601024 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:34.601345 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:34.601348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:34.601367 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:34.601638 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:34.601644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.602900 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.602905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:34.604969 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:34.604979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.606623 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.606629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:34.606800 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.606829 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 03:08:34.606953 (+ 124us) client_negotiation.cc:240] Negotiation successful
0807 03:08:34.607081 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":83,"spinlock_wait_cycles":19712,"thread_start_us":445,"threads_started":1}
I20260807 03:08:34.607681 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.600213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:40357 (local address 127.17.31.2:38483)
0807 03:08:34.600441 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:34.600448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:34.600943 (+ 495us) server_negotiation.cc:407] Connection header received
0807 03:08:34.601146 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:34.601150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:34.601206 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:34.601321 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:34.601791 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.602748 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.605103 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.606413 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.606555 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.606640 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:34.607271 (+ 631us) server_negotiation.cc:299] Negotiation successful
0807 03:08:34.607404 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 03:08:34.608117 18277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c2ccf20dd0492a8d8eb711c3e10aa1" candidate_uuid: "b7ea26ba7b464657a56109bc6e6280a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a9cd7cdc4e4e9c806a8256bede361e" is_pre_election: true
I20260807 03:08:34.608614 18418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c2ccf20dd0492a8d8eb711c3e10aa1" candidate_uuid: "b7ea26ba7b464657a56109bc6e6280a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f910cbe657fa4aadb5aeafdd547de0b5" is_pre_election: true
I20260807 03:08:34.608983 18277 raft_consensus.cc:2454] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P 71a9cd7cdc4e4e9c806a8256bede361e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7ea26ba7b464657a56109bc6e6280a0 in term 0.
I20260807 03:08:34.609216 18418 raft_consensus.cc:2454] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P f910cbe657fa4aadb5aeafdd547de0b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7ea26ba7b464657a56109bc6e6280a0 in term 0.
I20260807 03:08:34.609880 18486 leader_election.cc:302] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71a9cd7cdc4e4e9c806a8256bede361e, b7ea26ba7b464657a56109bc6e6280a0; no voters:
I20260807 03:08:34.610288 18626 raft_consensus.cc:2781] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:08:34.610510 18626 raft_consensus.cc:482] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:08:34.610631 18626 raft_consensus.cc:3037] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:34.612814 18626 raft_consensus.cc:504] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ea26ba7b464657a56109bc6e6280a0" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44117 } } peers { permanent_uuid: "71a9cd7cdc4e4e9c806a8256bede361e" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46705 } } peers { permanent_uuid: "f910cbe657fa4aadb5aeafdd547de0b5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 38483 } }
I20260807 03:08:34.613371 18626 leader_election.cc:288] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [CANDIDATE]: Term 1 election: Requested vote from peers 71a9cd7cdc4e4e9c806a8256bede361e (127.17.31.1:46705), f910cbe657fa4aadb5aeafdd547de0b5 (127.17.31.2:38483)
I20260807 03:08:34.613812 18277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c2ccf20dd0492a8d8eb711c3e10aa1" candidate_uuid: "b7ea26ba7b464657a56109bc6e6280a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a9cd7cdc4e4e9c806a8256bede361e"
I20260807 03:08:34.613812 18418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45c2ccf20dd0492a8d8eb711c3e10aa1" candidate_uuid: "b7ea26ba7b464657a56109bc6e6280a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f910cbe657fa4aadb5aeafdd547de0b5"
I20260807 03:08:34.614013 18418 raft_consensus.cc:3037] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P f910cbe657fa4aadb5aeafdd547de0b5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:34.614006 18277 raft_consensus.cc:3037] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P 71a9cd7cdc4e4e9c806a8256bede361e [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:34.617199 18418 raft_consensus.cc:2454] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P f910cbe657fa4aadb5aeafdd547de0b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7ea26ba7b464657a56109bc6e6280a0 in term 1.
I20260807 03:08:34.617201 18277 raft_consensus.cc:2454] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P 71a9cd7cdc4e4e9c806a8256bede361e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7ea26ba7b464657a56109bc6e6280a0 in term 1.
I20260807 03:08:34.617576 18487 leader_election.cc:302] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7ea26ba7b464657a56109bc6e6280a0, f910cbe657fa4aadb5aeafdd547de0b5; no voters:
I20260807 03:08:34.617899 18626 raft_consensus.cc:2781] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:08:34.618393 18626 raft_consensus.cc:686] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [term 1 LEADER]: Becoming Leader. State: Replica: b7ea26ba7b464657a56109bc6e6280a0, State: Running, Role: LEADER
I20260807 03:08:34.618875 18626 consensus_queue.cc:237] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ea26ba7b464657a56109bc6e6280a0" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44117 } } peers { permanent_uuid: "71a9cd7cdc4e4e9c806a8256bede361e" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46705 } } peers { permanent_uuid: "f910cbe657fa4aadb5aeafdd547de0b5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 38483 } }
I20260807 03:08:34.623795 18124 catalog_manager.cc:5495] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7ea26ba7b464657a56109bc6e6280a0 (127.17.31.3). New cstate: current_term: 1 leader_uuid: "b7ea26ba7b464657a56109bc6e6280a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7ea26ba7b464657a56109bc6e6280a0" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71a9cd7cdc4e4e9c806a8256bede361e" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f910cbe657fa4aadb5aeafdd547de0b5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 38483 } health_report { overall_health: UNKNOWN } } }
Aug 07 03:08:34 dist-test-slave-tm1g krb5kdc[18076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:08:34.673134 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.658472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38046 (local address 127.17.31.1:46705)
0807 03:08:34.658630 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:34.658634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:34.659202 (+ 568us) server_negotiation.cc:407] Connection header received
0807 03:08:34.659419 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:34.659423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:34.659509 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:34.659614 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:34.661153 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.661926 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.663283 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.663731 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.663867 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.668215 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:34.668228 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:34.668244 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:34.668279 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:34.670894 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:34.671482 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:34.671489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:34.671491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:34.671624 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:34.671968 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:34.671971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:34.671973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:34.672282 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:34.672493 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:34.672727 (+ 234us) server_negotiation.cc:299] Negotiation successful
0807 03:08:34.672872 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Aug 07 03:08:34 dist-test-slave-tm1g krb5kdc[18076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072114, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 03:08:34 dist-test-slave-tm1g krb5kdc[18076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072114, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:08:34.721072 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.707623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.17.31.1:46705)
0807 03:08:34.707816 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:34.707824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:34.708158 (+ 334us) server_negotiation.cc:407] Connection header received
0807 03:08:34.708357 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:34.708369 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:34.708432 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:34.708543 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:34.709874 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.710734 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.712038 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.712446 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.712587 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.716363 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:34.716368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:34.716371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:34.716400 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:34.719081 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:34.719601 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:34.719605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:34.719607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:34.719725 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:34.720062 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:34.720065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:34.720067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:34.720301 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:34.720483 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:34.720682 (+ 199us) server_negotiation.cc:299] Negotiation successful
0807 03:08:34.720835 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Aug 07 03:08:34 dist-test-slave-tm1g krb5kdc[18076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 03:08:34 dist-test-slave-tm1g krb5kdc[18076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:08:34.764770 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.751753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38066 (local address 127.17.31.1:46705)
0807 03:08:34.751978 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:34.751984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:34.752289 (+ 305us) server_negotiation.cc:407] Connection header received
0807 03:08:34.752491 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:34.752500 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:34.752565 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:34.752666 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:34.753969 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.754749 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.755916 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.756346 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.756473 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.760271 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:34.760275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:34.760279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:34.760309 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:34.762748 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:34.763309 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:34.763312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:34.763314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:34.763418 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:34.763707 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:34.763710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:34.763712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:34.763966 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:34.764146 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:34.764359 (+ 213us) server_negotiation.cc:299] Negotiation successful
0807 03:08:34.764528 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260807 03:08:34.766947 18234 tablet_service.cc:2908] Rejecting scan request for tablet 45c2ccf20dd0492a8d8eb711c3e10aa1: Uninitialized: clean time has not yet been initialized
I20260807 03:08:34.770116 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 18187
I20260807 03:08:34.783743 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.771298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.17.31.1:46705)
0807 03:08:34.771491 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:34.771511 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:34.771794 (+ 283us) server_negotiation.cc:407] Connection header received
0807 03:08:34.771929 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:34.771932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:34.771992 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:34.772067 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:34.773454 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.774102 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.775902 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.776345 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.776447 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.779002 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:34.779007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:34.779010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:34.779036 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:34.781410 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:34.782134 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:34.782138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:34.782140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:34.782243 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:34.782621 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:34.782624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:34.782626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:34.782862 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:34.783057 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:34.783350 (+ 293us) server_negotiation.cc:299] Negotiation successful
0807 03:08:34.783491 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 03:08:34.784313 18317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:08:34.884986 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 18187
I20260807 03:08:34.904214 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 18330
Aug 07 03:08:34 dist-test-slave-tm1g krb5kdc[18076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:08:34.918776 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:34.905020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.17.31.2:38483)
0807 03:08:34.905186 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:34.905191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:34.905229 (+ 38us) server_negotiation.cc:407] Connection header received
0807 03:08:34.905413 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:34.905417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:34.905506 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:34.905592 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:34.906795 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.907557 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.908823 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:34.909211 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:34.909297 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:34.913341 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:34.913356 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:34.913372 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:34.913401 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:34.916272 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:34.916868 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:34.916875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:34.916877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:34.917002 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:34.917315 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:34.917318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:34.917320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:34.917666 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:34.917897 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:34.918374 (+ 477us) server_negotiation.cc:299] Negotiation successful
0807 03:08:34.918534 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 03:08:34.919409 18458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:08:35.007292 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 18330
W20260807 03:08:35.024896 18487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.31.2:38483 (error 108)
I20260807 03:08:35.025151 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 18468
I20260807 03:08:35.026752 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:35.026281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:38483 (local address 127.17.31.3:33839)
0807 03:08:35.026468 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:35.026621 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.2:38483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260807 03:08:35.027097 18487 consensus_peers.cc:487] T 45c2ccf20dd0492a8d8eb711c3e10aa1 P b7ea26ba7b464657a56109bc6e6280a0 -> Peer f910cbe657fa4aadb5aeafdd547de0b5 (127.17.31.2:38483): Couldn't send request to peer f910cbe657fa4aadb5aeafdd547de0b5. Status: Network error: Client connection negotiation failed: client connection to 127.17.31.2:38483: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Aug 07 03:08:35 dist-test-slave-tm1g krb5kdc[18076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:08:35.040087 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:35.025935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44726 (local address 127.17.31.3:44117)
0807 03:08:35.026454 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:35.026460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:35.026487 (+ 27us) server_negotiation.cc:407] Connection header received
0807 03:08:35.026535 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:35.026539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:35.026600 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:35.026714 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:35.028018 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:35.028730 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:35.030031 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:35.030584 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:35.030714 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:35.034730 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:35.034740 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:35.034754 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:35.034782 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:35.037348 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:35.037987 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:35.037998 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:35.038006 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:35.038137 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:35.038596 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:35.038602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:35.038606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:35.038930 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:35.039227 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:35.039514 (+ 287us) server_negotiation.cc:299] Negotiation successful
0807 03:08:35.039748 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":501,"threads_started":1}
I20260807 03:08:35.040691 18597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:08:35.129002 18629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:35.128382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:46705 (local address 127.17.31.3:34363)
0807 03:08:35.128708 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:35.128789 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:46705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260807 03:08:35.129066 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 18468
I20260807 03:08:35.146428 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 18089
I20260807 03:08:35.147545 18155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:08:35.254197 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 18089
2026-08-07T03:08:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4181 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:08:35 dist-test-slave-tm1g krb5kdc[18652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:08:35 dist-test-slave-tm1g krb5kdc[18652](info): set up 2 sockets
Aug 07 03:08:35 dist-test-slave-tm1g krb5kdc[18652](info): commencing operation
krb5kdc: starting...
W20260807 03:08:36.363739 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:08:36 dist-test-slave-tm1g krb5kdc[18652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:08:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:08:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:08:36.539790 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:44145
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:36347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:44145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:36.857491 18665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:36.857813 18665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:36.857986 18665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:36.865037 18665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:08:36.865175 18665 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 03:08:36.865265 18665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:36.865401 18665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:36.865485 18665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:08:36.865595 18665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:08:36.875228 18665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:36347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:44145
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:44145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:36.876474 18665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:36.877806 18665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:36.890117 18670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:36.890090 18671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:36.890902 18673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:36.891182 18665 server_base.cc:1034] running on GCE node
I20260807 03:08:36.891960 18665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:36.893329 18665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:36.894568 18665 hybrid_clock.cc:648] HybridClock initialized: now 1786072116894503 us; error 41 us; skew 500 ppm
I20260807 03:08:36.894878 18665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:36 dist-test-slave-tm1g krb5kdc[18652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072116, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:36.897764 18665 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:08:36.899209 18665 webserver.cc:460] Webserver started at http://127.17.31.62:43189/ using document root <none> and password file <none>
I20260807 03:08:36.899892 18665 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:36.900024 18665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:36.900341 18665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:36.903242 18665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "9f7669eb1cb44212b2a0440c79c7e960"
format_stamp: "Formatted at 2026-08-07 03:08:36 on dist-test-slave-tm1g"
server_key: "afd6a764631cb3e52f366203ccb8eac0"
server_key_iv: "ffb5019e226ed051f77f8c59d6ea478e"
server_key_version: "encryptionkey@0"
I20260807 03:08:36.904012 18665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "9f7669eb1cb44212b2a0440c79c7e960"
format_stamp: "Formatted at 2026-08-07 03:08:36 on dist-test-slave-tm1g"
server_key: "afd6a764631cb3e52f366203ccb8eac0"
server_key_iv: "ffb5019e226ed051f77f8c59d6ea478e"
server_key_version: "encryptionkey@0"
I20260807 03:08:36.909056 18665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:08:36.912297 18680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:36.913441 18665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 03:08:36.913633 18665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "9f7669eb1cb44212b2a0440c79c7e960"
format_stamp: "Formatted at 2026-08-07 03:08:36 on dist-test-slave-tm1g"
server_key: "afd6a764631cb3e52f366203ccb8eac0"
server_key_iv: "ffb5019e226ed051f77f8c59d6ea478e"
server_key_version: "encryptionkey@0"
I20260807 03:08:36.913844 18665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:36.932132 18665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:36.935922 18665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:36.936182 18665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:36.957365 18665 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:44145
I20260807 03:08:36.957391 18745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:44145 every 8 connection(s)
I20260807 03:08:36.958747 18665 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:08:36.962177 18746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:36.965238 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 18665
I20260807 03:08:36.965516 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:08:36.966387 17532 external_mini_cluster.cc:1428] Setting key 85fc8d4e493699cf051c4829e692c0ea
I20260807 03:08:36.977931 18746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960: Bootstrap starting.
I20260807 03:08:36.981474 18746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:36.982501 18746 log.cc:824] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960: Log is configured to *not* fsync() on all Append() calls
Aug 07 03:08:36 dist-test-slave-tm1g krb5kdc[18652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:36.985391 18746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960: No bootstrap required, opened a new log
I20260807 03:08:36.991710 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:36.969366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.17.31.62:44145)
0807 03:08:36.971420 (+ 2054us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:36.971436 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:36.971488 (+ 52us) server_negotiation.cc:407] Connection header received
0807 03:08:36.973803 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:36.973834 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:36.974349 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:36.974778 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:36.976031 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:36.977930 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:36.979215 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:36.980291 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:36.980470 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:36.984730 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:36.984738 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:36.984751 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:36.984782 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:36.987769 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:36.988555 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:36.988561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:36.988567 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:36.988717 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:36.989111 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:36.989114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:36.989116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:36.989603 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:36.989822 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:36.990272 (+ 450us) server_negotiation.cc:299] Negotiation successful
0807 03:08:36.990706 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1800,"thread_start_us":885,"threads_started":1}
I20260807 03:08:36.993803 18746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7669eb1cb44212b2a0440c79c7e960" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44145 } }
I20260807 03:08:36.994139 18746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:36.994237 18746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7669eb1cb44212b2a0440c79c7e960, State: Initialized, Role: FOLLOWER
I20260807 03:08:36.994799 18746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7669eb1cb44212b2a0440c79c7e960" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44145 } }
I20260807 03:08:36.995034 18746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:08:36.995174 18746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:08:36.995332 18746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:36.997315 18746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7669eb1cb44212b2a0440c79c7e960" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44145 } }
I20260807 03:08:36.997785 18746 leader_election.cc:302] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f7669eb1cb44212b2a0440c79c7e960; no voters:
I20260807 03:08:36.998376 18746 leader_election.cc:288] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:08:36.998517 18752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:08:36.999279 18752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [term 1 LEADER]: Becoming Leader. State: Replica: 9f7669eb1cb44212b2a0440c79c7e960, State: Running, Role: LEADER
I20260807 03:08:36.999804 18752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7669eb1cb44212b2a0440c79c7e960" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44145 } }
I20260807 03:08:37.000375 18746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:08:37.002445 18753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f7669eb1cb44212b2a0440c79c7e960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7669eb1cb44212b2a0440c79c7e960" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44145 } } }
I20260807 03:08:37.002794 18753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [sys.catalog]: This master's current role is: LEADER
I20260807 03:08:37.002924 18754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f7669eb1cb44212b2a0440c79c7e960. Latest consensus state: current_term: 1 leader_uuid: "9f7669eb1cb44212b2a0440c79c7e960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7669eb1cb44212b2a0440c79c7e960" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44145 } } }
I20260807 03:08:37.003230 18754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960 [sys.catalog]: This master's current role is: LEADER
I20260807 03:08:37.004540 18757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:08:37.009389 18757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:08:37.018582 18757 catalog_manager.cc:1347] Generated new cluster ID: be46ae12a0e34ebd88bc02572cb99e0e
I20260807 03:08:37.018684 18757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:08:37.040074 18757 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:08:37.041872 18757 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:08:37.049822 18757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960: Generated new TSK 0
I20260807 03:08:37.050809 18757 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:08:37.096105 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44145
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:36347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:37.414516 18773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:37.414762 18773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:37.414937 18773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:37.422013 18773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:37.422132 18773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:37.422283 18773 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 03:08:37.422441 18773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:08:37.431694 18773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:36347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.31.62:44145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:37.432878 18773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:37.434213 18773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:37.446707 18778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:37.446764 18779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:37.446985 18781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:37.447649 18773 server_base.cc:1034] running on GCE node
I20260807 03:08:37.448220 18773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:37.449055 18773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:37.450294 18773 hybrid_clock.cc:648] HybridClock initialized: now 1786072117450240 us; error 48 us; skew 500 ppm
I20260807 03:08:37.450630 18773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:37 dist-test-slave-tm1g krb5kdc[18652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072117, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:37.453567 18773 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:08:37.455076 18773 webserver.cc:460] Webserver started at http://127.17.31.1:37733/ using document root <none> and password file <none>
I20260807 03:08:37.455770 18773 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:37.455860 18773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:37.456130 18773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:37.458880 18773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "d7d9f7441a9a4784ab803210fe1ce9d3"
format_stamp: "Formatted at 2026-08-07 03:08:37 on dist-test-slave-tm1g"
server_key: "559afba17782c85857ecfd10c84770a6"
server_key_iv: "18a94125c72d9ce59e20a4ecb4e3e201"
server_key_version: "encryptionkey@0"
I20260807 03:08:37.459627 18773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "d7d9f7441a9a4784ab803210fe1ce9d3"
format_stamp: "Formatted at 2026-08-07 03:08:37 on dist-test-slave-tm1g"
server_key: "559afba17782c85857ecfd10c84770a6"
server_key_iv: "18a94125c72d9ce59e20a4ecb4e3e201"
server_key_version: "encryptionkey@0"
I20260807 03:08:37.464366 18773 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 03:08:37.467563 18788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:37.468750 18773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:08:37.468910 18773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "d7d9f7441a9a4784ab803210fe1ce9d3"
format_stamp: "Formatted at 2026-08-07 03:08:37 on dist-test-slave-tm1g"
server_key: "559afba17782c85857ecfd10c84770a6"
server_key_iv: "18a94125c72d9ce59e20a4ecb4e3e201"
server_key_version: "encryptionkey@0"
I20260807 03:08:37.469122 18773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:37.482903 18773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:37.486615 18773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:37.486866 18773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:37.489411 18773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:37.489519 18773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:37.489673 18773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:37.489779 18773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 03:08:37 dist-test-slave-tm1g krb5kdc[18652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072117, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:37.512153 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:37.490094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:44881 (local address 127.17.31.62:44145)
0807 03:08:37.490291 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:37.490316 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:37.491913 (+ 1597us) server_negotiation.cc:407] Connection header received
0807 03:08:37.493856 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:37.493861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:37.493944 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:37.494064 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:37.496255 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:37.497548 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:37.499981 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:37.500625 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:37.500758 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:37.505176 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:37.505186 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:37.505194 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:37.505250 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:37.508729 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:37.509676 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:37.509682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:37.509686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:37.509877 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:37.510353 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:37.510357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:37.510361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:37.510665 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:37.510817 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:37.511611 (+ 794us) server_negotiation.cc:299] Negotiation successful
0807 03:08:37.511822 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 03:08:37.513633 18797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:37.490968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44145 (local address 127.17.31.1:44881)
0807 03:08:37.491764 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:37.491807 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:37.493597 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:37.494287 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:37.494294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:37.495058 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:37.496037 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:37.496065 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:37.497744 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:37.497750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:37.499796 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:37.499812 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:37.500885 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:37.500892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:37.501105 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:37.501850 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:37.501887 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:37.504884 (+ 2997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:37.508883 (+ 3999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:37.508888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:37.508898 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:37.509546 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:37.509972 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:37.509977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:37.509981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:37.510205 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:37.510820 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:37.510829 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:37.511200 (+ 371us) client_negotiation.cc:769] Sending connection context
0807 03:08:37.511514 (+ 314us) client_negotiation.cc:240] Negotiation successful
0807 03:08:37.511974 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":350,"threads_started":1}
I20260807 03:08:37.534821 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:37.519822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44145 (local address 127.17.31.1:40745)
0807 03:08:37.520546 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:37.520571 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:37.520741 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:37.521068 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:37.521075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:37.521778 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:37.522249 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:37.522264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:37.523499 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:37.523504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:37.525207 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:37.525219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:37.526178 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:37.526184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:37.526473 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:37.527216 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:37.527233 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:37.528185 (+ 952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:37.531659 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:37.531663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:37.531665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:37.532273 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:37.532728 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:37.532736 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:37.532744 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:37.532897 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:37.533526 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:37.533530 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:37.533805 (+ 275us) client_negotiation.cc:769] Sending connection context
0807 03:08:37.533986 (+ 181us) client_negotiation.cc:240] Negotiation successful
0807 03:08:37.534280 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":381,"threads_started":1}
I20260807 03:08:37.535684 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:37.519785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:40745 (local address 127.17.31.62:44145)
0807 03:08:37.519949 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:37.519956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:37.520675 (+ 719us) server_negotiation.cc:407] Connection header received
0807 03:08:37.520872 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:37.520875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:37.520929 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:37.521005 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:37.522420 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:37.523366 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:37.525354 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:37.525958 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:37.526116 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:37.528371 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:37.528375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:37.528379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:37.528406 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:37.531481 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:37.532396 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:37.532404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:37.532407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:37.532579 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:37.533013 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:37.533018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:37.533022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:37.533403 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:37.533556 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:37.535133 (+ 1577us) server_negotiation.cc:299] Negotiation successful
0807 03:08:37.535381 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 03:08:37.536422 18773 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:45495
I20260807 03:08:37.536439 18913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:45495 every 8 connection(s)
I20260807 03:08:37.537624 18773 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:08:37.539760 18914 heartbeater.cc:347] Connected to a master server at 127.17.31.62:44145
I20260807 03:08:37.539992 18914 heartbeater.cc:464] Registering TS with master...
I20260807 03:08:37.540601 18914 heartbeater.cc:511] Master 127.17.31.62:44145 requested a full tablet report, sending...
I20260807 03:08:37.542419 18700 ts_manager.cc:194] Registered new tserver with Master: d7d9f7441a9a4784ab803210fe1ce9d3 (127.17.31.1:45495)
I20260807 03:08:37.543694 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 18773
I20260807 03:08:37.543783 18700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:44881
I20260807 03:08:37.543956 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:08:37.544668 17532 external_mini_cluster.cc:1428] Setting key 7fb0d18b5da8e2727dc6d73ae26d5a8c
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:08:37.583230 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44145
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:36347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:37.902282 18918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:37.902622 18918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:37.902791 18918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:37.909890 18918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:37.910027 18918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:37.910176 18918 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 03:08:37.910357 18918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:08:37.919952 18918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:36347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.31.62:44145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:37.921149 18918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:37.922518 18918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:37.935801 18923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:37.935743 18924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:37.935715 18926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:37.936581 18918 server_base.cc:1034] running on GCE node
I20260807 03:08:37.937281 18918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:37.938125 18918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:37.939345 18918 hybrid_clock.cc:648] HybridClock initialized: now 1786072117939254 us; error 48 us; skew 500 ppm
I20260807 03:08:37.939622 18918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:37 dist-test-slave-tm1g krb5kdc[18652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072117, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:37.942613 18918 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:08:37.943977 18918 webserver.cc:460] Webserver started at http://127.17.31.2:44315/ using document root <none> and password file <none>
I20260807 03:08:37.944662 18918 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:37.944793 18918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:37.945112 18918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:37.947952 18918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "9b60a62a8bc0455d89ca6d0870a41bd5"
format_stamp: "Formatted at 2026-08-07 03:08:37 on dist-test-slave-tm1g"
server_key: "cbdd0e3ad5dc7d6a2d0a043b724e2fa3"
server_key_iv: "33e479e569af22b0aac1f5e2c5564abc"
server_key_version: "encryptionkey@0"
I20260807 03:08:37.948711 18918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "9b60a62a8bc0455d89ca6d0870a41bd5"
format_stamp: "Formatted at 2026-08-07 03:08:37 on dist-test-slave-tm1g"
server_key: "cbdd0e3ad5dc7d6a2d0a043b724e2fa3"
server_key_iv: "33e479e569af22b0aac1f5e2c5564abc"
server_key_version: "encryptionkey@0"
I20260807 03:08:37.953621 18918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:08:37.957270 18933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:37.958676 18918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:08:37.958823 18918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "9b60a62a8bc0455d89ca6d0870a41bd5"
format_stamp: "Formatted at 2026-08-07 03:08:37 on dist-test-slave-tm1g"
server_key: "cbdd0e3ad5dc7d6a2d0a043b724e2fa3"
server_key_iv: "33e479e569af22b0aac1f5e2c5564abc"
server_key_version: "encryptionkey@0"
I20260807 03:08:37.959024 18918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:37.979247 18918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:37.982812 18918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:37.983067 18918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:37.985530 18918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:37.985651 18918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:37.985805 18918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:37.985883 18918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 03:08:37 dist-test-slave-tm1g krb5kdc[18652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072117, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:38.007133 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:37.985828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:40229 (local address 127.17.31.62:44145)
0807 03:08:37.986057 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:37.986063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:37.987431 (+ 1368us) server_negotiation.cc:407] Connection header received
0807 03:08:37.989445 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:37.989451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:37.989538 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:37.989639 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:37.991594 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:37.992950 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:37.995390 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:37.995974 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:37.996114 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.000056 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:38.000061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:38.000064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:38.000096 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:38.003759 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:38.004587 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:38.004596 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:38.004602 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:38.004780 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:38.005178 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:38.005186 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:38.005192 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:38.005636 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:38.005807 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:38.006410 (+ 603us) server_negotiation.cc:299] Negotiation successful
0807 03:08:38.006679 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 03:08:38.008116 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:37.986435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44145 (local address 127.17.31.2:40229)
0807 03:08:37.987268 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:37.987311 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:37.989177 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:37.989840 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:37.989847 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:37.990593 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:37.991378 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:37.991415 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:37.993149 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:37.993156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:37.995234 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:37.995245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:37.996232 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:37.996240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:37.996435 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:37.997142 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:37.997166 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:37.999793 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:38.003912 (+ 4119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:38.003924 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:38.003942 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:38.004455 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:38.004891 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:38.004894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:38.004896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:38.005060 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:38.005768 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:38.005773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:38.006035 (+ 262us) client_negotiation.cc:769] Sending connection context
0807 03:08:38.006235 (+ 200us) client_negotiation.cc:240] Negotiation successful
0807 03:08:38.006844 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":362,"threads_started":1}
I20260807 03:08:38.032997 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.017257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44145 (local address 127.17.31.2:48671)
0807 03:08:38.018013 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:38.018043 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:38.018222 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:38.018588 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:38.018595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:38.019357 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:38.019821 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.019837 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.020818 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.020826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.022919 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.022939 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.023896 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.023909 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.024218 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.024935 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:38.024964 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:38.026390 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:38.029830 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:38.029838 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:38.029845 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:38.030517 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:38.030900 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:38.030905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:38.030909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:38.031104 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:38.031847 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:38.031854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:38.032047 (+ 193us) client_negotiation.cc:769] Sending connection context
0807 03:08:38.032154 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 03:08:38.032560 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":388,"threads_started":1}
I20260807 03:08:38.033835 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.017303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:48671 (local address 127.17.31.62:44145)
0807 03:08:38.017517 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:38.017525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:38.018120 (+ 595us) server_negotiation.cc:407] Connection header received
0807 03:08:38.018396 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:38.018399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:38.018454 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:38.018526 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:38.019992 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.020697 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.023086 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.023582 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.023725 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.026610 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:38.026616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:38.026621 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:38.026661 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:38.029704 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:38.030658 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:38.030663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:38.030667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:38.030784 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:38.031267 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:38.031273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:38.031279 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:38.031677 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:38.033347 (+ 1670us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:38.033438 (+ 91us) server_negotiation.cc:299] Negotiation successful
0807 03:08:38.033610 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 03:08:38.044637 18918 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:36411
I20260807 03:08:38.044663 19060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:36411 every 8 connection(s)
I20260807 03:08:38.045819 18918 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:08:38.047816 19061 heartbeater.cc:347] Connected to a master server at 127.17.31.62:44145
I20260807 03:08:38.048034 19061 heartbeater.cc:464] Registering TS with master...
I20260807 03:08:38.048643 19061 heartbeater.cc:511] Master 127.17.31.62:44145 requested a full tablet report, sending...
I20260807 03:08:38.048988 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 18918
I20260807 03:08:38.049182 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:08:38.049742 17532 external_mini_cluster.cc:1428] Setting key e1f72410fff6574007202e1158640589
I20260807 03:08:38.050045 18700 ts_manager.cc:194] Registered new tserver with Master: 9b60a62a8bc0455d89ca6d0870a41bd5 (127.17.31.2:36411)
I20260807 03:08:38.051381 18700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:40229
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:08:38.090772 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44145
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:36347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:38.408910 19066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:38.409204 19066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:38.409380 19066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:38.416778 19066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:38.416909 19066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:38.417053 19066 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 03:08:38.417186 19066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:08:38.426637 19066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:36347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.31.62:44145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:38.427904 19066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:38.429217 19066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:38.441083 19071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:38.441992 19072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:38.442605 19074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:38.443149 19066 server_base.cc:1034] running on GCE node
I20260807 03:08:38.443728 19066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:38.444591 19066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:38.445829 19066 hybrid_clock.cc:648] HybridClock initialized: now 1786072118445749 us; error 50 us; skew 500 ppm
I20260807 03:08:38.446115 19066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:38 dist-test-slave-tm1g krb5kdc[18652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072118, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:38.449102 19066 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:08:38.450428 19066 webserver.cc:460] Webserver started at http://127.17.31.3:36477/ using document root <none> and password file <none>
I20260807 03:08:38.451067 19066 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:38.451151 19066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:38.451419 19066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:38.454095 19066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "93f71254483b46f29e00456f62510c25"
format_stamp: "Formatted at 2026-08-07 03:08:38 on dist-test-slave-tm1g"
server_key: "43bed1b4e618edbe34c9f0dc7f610d8e"
server_key_iv: "25996891f960842cedac770a265093b8"
server_key_version: "encryptionkey@0"
I20260807 03:08:38.454885 19066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "93f71254483b46f29e00456f62510c25"
format_stamp: "Formatted at 2026-08-07 03:08:38 on dist-test-slave-tm1g"
server_key: "43bed1b4e618edbe34c9f0dc7f610d8e"
server_key_iv: "25996891f960842cedac770a265093b8"
server_key_version: "encryptionkey@0"
I20260807 03:08:38.459846 19066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 03:08:38.463244 19081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:38.464540 19066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 03:08:38.464704 19066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "93f71254483b46f29e00456f62510c25"
format_stamp: "Formatted at 2026-08-07 03:08:38 on dist-test-slave-tm1g"
server_key: "43bed1b4e618edbe34c9f0dc7f610d8e"
server_key_iv: "25996891f960842cedac770a265093b8"
server_key_version: "encryptionkey@0"
I20260807 03:08:38.464908 19066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:38.488312 19066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:38.491964 19066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:38.492250 19066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:38.494796 19066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:38.494907 19066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:38.495045 19066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:38.495121 19066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 03:08:38 dist-test-slave-tm1g krb5kdc[18652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072118, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:38.519950 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.495471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:42875 (local address 127.17.31.62:44145)
0807 03:08:38.495706 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:38.495712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:38.497148 (+ 1436us) server_negotiation.cc:407] Connection header received
0807 03:08:38.499513 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:38.499522 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:38.499627 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:38.499709 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:38.501959 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.503315 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.505717 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.506373 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.506504 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.511776 (+ 5272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:38.511785 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:38.511793 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:38.511842 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:38.515871 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:38.516744 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:38.516750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:38.516754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:38.516925 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:38.517764 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:38.517770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:38.517775 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:38.518089 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:38.518255 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:38.519323 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0807 03:08:38.519572 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 03:08:38.521507 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.496097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44145 (local address 127.17.31.3:42875)
0807 03:08:38.496953 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:38.497007 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:38.499152 (+ 2145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:38.500279 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:38.500289 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:38.501108 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:38.501771 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.501787 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.503483 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.503490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.505514 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.505526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.506615 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.506627 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.506843 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.507544 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:38.507596 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:38.511537 (+ 3941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:38.516006 (+ 4469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:38.516018 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:38.516038 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:38.516598 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:38.517099 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:38.517107 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:38.517112 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:38.517576 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:38.518424 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:38.518434 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:38.518855 (+ 421us) client_negotiation.cc:769] Sending connection context
0807 03:08:38.519248 (+ 393us) client_negotiation.cc:240] Negotiation successful
0807 03:08:38.519812 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":387,"threads_started":1}
I20260807 03:08:38.543413 19066 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:44737
I20260807 03:08:38.543455 19204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:44737 every 8 connection(s)
I20260807 03:08:38.544966 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.529531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44145 (local address 127.17.31.3:60197)
0807 03:08:38.530128 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:38.530142 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:38.530289 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:38.530660 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:38.530664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:38.531412 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:38.531889 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.531909 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.533202 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.533212 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.535227 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.535242 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.536393 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.536400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.536566 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.537245 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:38.537278 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:38.538277 (+ 999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:38.542087 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:38.542095 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:38.542102 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:38.542776 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:38.543316 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:38.543322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:38.543326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:38.543516 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:38.544120 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:38.544123 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:38.544269 (+ 146us) client_negotiation.cc:769] Sending connection context
0807 03:08:38.544373 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 03:08:38.544610 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":350,"threads_started":1}
I20260807 03:08:38.544977 19066 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:08:38.545185 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.529546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:60197 (local address 127.17.31.62:44145)
0807 03:08:38.529680 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:38.529685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:38.530228 (+ 543us) server_negotiation.cc:407] Connection header received
0807 03:08:38.530465 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:38.530470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:38.530538 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:38.530632 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:38.532041 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.533073 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.535378 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.536029 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.536148 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.538508 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:38.538514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:38.538518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:38.538557 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:38.541927 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:38.542959 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:38.542966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:38.542969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:38.543162 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:38.543658 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:38.543663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:38.543666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:38.544000 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:38.544154 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:38.544558 (+ 404us) server_negotiation.cc:299] Negotiation successful
0807 03:08:38.544768 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 03:08:38.547755 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 19066
I20260807 03:08:38.547967 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:08:38.548197 18914 heartbeater.cc:503] Master 127.17.31.62:44145 was elected leader, sending a full tablet report...
I20260807 03:08:38.548543 17532 external_mini_cluster.cc:1428] Setting key 6994fb9ecc32c7941ee3daf6554b27a4
I20260807 03:08:38.549484 19205 heartbeater.cc:347] Connected to a master server at 127.17.31.62:44145
I20260807 03:08:38.549741 19205 heartbeater.cc:464] Registering TS with master...
I20260807 03:08:38.550468 19205 heartbeater.cc:511] Master 127.17.31.62:44145 requested a full tablet report, sending...
I20260807 03:08:38.551872 18700 ts_manager.cc:194] Registered new tserver with Master: 93f71254483b46f29e00456f62510c25 (127.17.31.3:44737)
I20260807 03:08:38.552606 18700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:42875
I20260807 03:08:38.563812 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 03:08:38 dist-test-slave-tm1g krb5kdc[18652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072118, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 03:08:38 dist-test-slave-tm1g krb5kdc[18652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072118, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:38.609246 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.596349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.17.31.62:44145)
0807 03:08:38.596598 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:38.596605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:38.596736 (+ 131us) server_negotiation.cc:407] Connection header received
0807 03:08:38.596887 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:38.596891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:38.596969 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:38.597057 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:38.598379 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.599109 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.600371 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.600794 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.600890 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.604682 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:38.604687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:38.604690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:38.604723 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:38.607103 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:38.607650 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:38.607654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:38.607657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:38.607780 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:38.608096 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:38.608099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:38.608101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:38.608337 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:38.608515 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:38.608738 (+ 223us) server_negotiation.cc:299] Negotiation successful
0807 03:08:38.608907 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 03:08:38.613175 18700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:08:38.620309 18700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:08:38.643141 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.632042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.17.31.3:44737)
0807 03:08:38.633280 (+ 1238us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:38.633289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:38.633755 (+ 466us) server_negotiation.cc:407] Connection header received
0807 03:08:38.634164 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:38.634172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:38.634430 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:38.634626 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:38.635999 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.637790 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.640434 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.641914 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.642055 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.642321 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:38.642618 (+ 297us) server_negotiation.cc:299] Negotiation successful
0807 03:08:38.642782 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1059,"thread_start_us":1415,"threads_started":1}
I20260807 03:08:38.643213 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.631779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:44737 (local address 127.0.0.1:48714)
0807 03:08:38.632511 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:38.633649 (+ 1138us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:38.633903 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:38.635068 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:38.635075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:38.635110 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:38.635711 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.635727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.637930 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.637937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.640242 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.640256 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.642134 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.642143 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.642371 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.642421 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 03:08:38.642579 (+ 158us) client_negotiation.cc:240] Negotiation successful
0807 03:08:38.642761 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":57,"spinlock_wait_cycles":1920,"thread_start_us":356,"threads_started":1}
I20260807 03:08:38.643872 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.632373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42400 (local address 127.17.31.1:45495)
0807 03:08:38.634019 (+ 1646us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:38.634029 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:38.634054 (+ 25us) server_negotiation.cc:407] Connection header received
0807 03:08:38.634152 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:38.634163 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:38.634441 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:38.634658 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:38.637402 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.638688 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.641073 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.642093 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.642218 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.642458 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:38.643384 (+ 926us) server_negotiation.cc:299] Negotiation successful
0807 03:08:38.643568 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1492,"thread_start_us":1701,"threads_started":1}
I20260807 03:08:38.644418 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.631928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45495 (local address 127.0.0.1:42400)
0807 03:08:38.633703 (+ 1775us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:38.633725 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:38.633903 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:38.634949 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:38.634960 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:38.634998 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:38.637231 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.637242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.638826 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.638831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.640919 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.640930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.642599 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.642604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.642756 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.642784 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 03:08:38.642868 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 03:08:38.644068 (+ 1200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1635,"mutex_wait_us":1669,"spinlock_wait_cycles":10752,"thread_start_us":1505,"threads_started":1}
I20260807 03:08:38.645503 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.634660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.17.31.2:36411)
0807 03:08:38.637159 (+ 2499us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:38.637167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:38.637193 (+ 26us) server_negotiation.cc:407] Connection header received
0807 03:08:38.637291 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:38.637301 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:38.637476 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:38.637665 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:38.638289 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.639626 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.642365 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.644565 (+ 2200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.644692 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.644886 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:38.645068 (+ 182us) server_negotiation.cc:299] Negotiation successful
0807 03:08:38.645218 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2345,"thread_start_us":2232,"threads_started":1}
I20260807 03:08:38.645491 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.634267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:36411 (local address 127.0.0.1:35446)
0807 03:08:38.636823 (+ 2556us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:38.636844 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:38.636955 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:38.637643 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:38.637647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:38.637670 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:38.638086 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.638101 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.639805 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.639810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.642148 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.642162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.644760 (+ 2598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.644765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.644879 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.644900 (+ 21us) client_negotiation.cc:769] Sending connection context
0807 03:08:38.645002 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 03:08:38.645129 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2444,"mutex_wait_us":60,"thread_start_us":2126,"threads_started":1}
I20260807 03:08:38.647603 19133 tablet_service.cc:1467] Processing CreateTablet for tablet 91378dd62f254632bd3ce0c30499c2ba (DEFAULT_TABLE table=test-table [id=ff47e5ce14de4ee6bd3c2965acfb2494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:38.648224 18840 tablet_service.cc:1467] Processing CreateTablet for tablet 91378dd62f254632bd3ce0c30499c2ba (DEFAULT_TABLE table=test-table [id=ff47e5ce14de4ee6bd3c2965acfb2494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:38.648759 18986 tablet_service.cc:1467] Processing CreateTablet for tablet 91378dd62f254632bd3ce0c30499c2ba (DEFAULT_TABLE table=test-table [id=ff47e5ce14de4ee6bd3c2965acfb2494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:38.649501 18840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91378dd62f254632bd3ce0c30499c2ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:38.649528 19133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91378dd62f254632bd3ce0c30499c2ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:38.649917 18986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91378dd62f254632bd3ce0c30499c2ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:38.664124 19225 tablet_bootstrap.cc:492] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3: Bootstrap starting.
I20260807 03:08:38.666424 19224 tablet_bootstrap.cc:492] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5: Bootstrap starting.
I20260807 03:08:38.668270 19225 tablet_bootstrap.cc:654] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:38.669472 19225 log.cc:824] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3: Log is configured to *not* fsync() on all Append() calls
I20260807 03:08:38.669973 19226 tablet_bootstrap.cc:492] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25: Bootstrap starting.
I20260807 03:08:38.670521 19224 tablet_bootstrap.cc:654] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:38.671895 19224 log.cc:824] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5: Log is configured to *not* fsync() on all Append() calls
I20260807 03:08:38.672690 19225 tablet_bootstrap.cc:492] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3: No bootstrap required, opened a new log
I20260807 03:08:38.673051 19225 ts_tablet_manager.cc:1397] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260807 03:08:38.673632 19226 tablet_bootstrap.cc:654] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:38.674863 19226 log.cc:824] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25: Log is configured to *not* fsync() on all Append() calls
I20260807 03:08:38.675225 19224 tablet_bootstrap.cc:492] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5: No bootstrap required, opened a new log
I20260807 03:08:38.675529 19224 ts_tablet_manager.cc:1397] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 03:08:38.677712 19226 tablet_bootstrap.cc:492] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25: No bootstrap required, opened a new log
I20260807 03:08:38.677964 19226 ts_tablet_manager.cc:1397] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 03:08:38.680591 19225 raft_consensus.cc:348] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } }
I20260807 03:08:38.681150 19225 raft_consensus.cc:374] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:38.681304 19225 raft_consensus.cc:729] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d9f7441a9a4784ab803210fe1ce9d3, State: Initialized, Role: FOLLOWER
I20260807 03:08:38.682211 19225 consensus_queue.cc:260] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } }
I20260807 03:08:38.683763 19225 ts_tablet_manager.cc:1428] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260807 03:08:38.686478 19224 raft_consensus.cc:348] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } }
I20260807 03:08:38.686951 19224 raft_consensus.cc:374] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:38.687062 19224 raft_consensus.cc:729] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b60a62a8bc0455d89ca6d0870a41bd5, State: Initialized, Role: FOLLOWER
I20260807 03:08:38.687230 19226 raft_consensus.cc:348] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } }
I20260807 03:08:38.687668 19224 consensus_queue.cc:260] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } }
I20260807 03:08:38.688009 19226 raft_consensus.cc:374] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:38.688126 19226 raft_consensus.cc:729] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f71254483b46f29e00456f62510c25, State: Initialized, Role: FOLLOWER
I20260807 03:08:38.688731 19061 heartbeater.cc:503] Master 127.17.31.62:44145 was elected leader, sending a full tablet report...
I20260807 03:08:38.688869 19226 consensus_queue.cc:260] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } }
I20260807 03:08:38.689442 19224 ts_tablet_manager.cc:1428] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260807 03:08:38.690622 19205 heartbeater.cc:503] Master 127.17.31.62:44145 was elected leader, sending a full tablet report...
I20260807 03:08:38.691203 19226 ts_tablet_manager.cc:1428] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260807 03:08:38.787742 19230 raft_consensus.cc:482] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:08:38.788061 19230 raft_consensus.cc:504] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } }
I20260807 03:08:38.789945 19230 leader_election.cc:288] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93f71254483b46f29e00456f62510c25 (127.17.31.3:44737), 9b60a62a8bc0455d89ca6d0870a41bd5 (127.17.31.2:36411)
W20260807 03:08:38.790676 18915 tablet.cc:2369] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:08:38.797180 19206 tablet.cc:2369] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:08:38.797807 19062 tablet.cc:2369] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:08:38.799583 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.790701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:37531 (local address 127.17.31.2:36411)
0807 03:08:38.790923 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:38.790929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:38.791302 (+ 373us) server_negotiation.cc:407] Connection header received
0807 03:08:38.791663 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:38.791671 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:38.791792 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:38.791912 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:38.792677 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.793977 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.797196 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.798522 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.798669 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.798795 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:38.799202 (+ 407us) server_negotiation.cc:299] Negotiation successful
0807 03:08:38.799317 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 03:08:38.799599 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.790233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:44041 (local address 127.17.31.3:44737)
0807 03:08:38.790494 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:38.790501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:38.791638 (+ 1137us) server_negotiation.cc:407] Connection header received
0807 03:08:38.791868 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:38.791876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:38.791959 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:38.792052 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:38.792766 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.793977 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.797176 (+ 3199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.798566 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.798737 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.798852 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:38.799167 (+ 315us) server_negotiation.cc:299] Negotiation successful
0807 03:08:38.799304 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 03:08:38.799674 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.790336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:44737 (local address 127.17.31.1:44041)
0807 03:08:38.791545 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:38.791568 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:38.791720 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:38.792085 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:38.792091 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:38.792128 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:38.792579 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.792592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.794124 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.794128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.797008 (+ 2880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.797019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.798832 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.798839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.799003 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.799032 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 03:08:38.799107 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 03:08:38.799255 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"mutex_wait_us":245,"thread_start_us":917,"threads_started":1}
I20260807 03:08:38.799772 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.790232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:36411 (local address 127.17.31.1:37531)
0807 03:08:38.791186 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:38.791210 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:38.791450 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:38.791929 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:38.791932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:38.791956 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:38.792503 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.792516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.794122 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.794128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.797004 (+ 2876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.797019 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.798787 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.798795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.799012 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.799043 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 03:08:38.799164 (+ 121us) client_negotiation.cc:240] Negotiation successful
0807 03:08:38.799304 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":213,"thread_start_us":702,"threads_started":1}
I20260807 03:08:38.800765 19154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91378dd62f254632bd3ce0c30499c2ba" candidate_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f71254483b46f29e00456f62510c25" is_pre_election: true
I20260807 03:08:38.800916 19013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91378dd62f254632bd3ce0c30499c2ba" candidate_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" is_pre_election: true
I20260807 03:08:38.801458 19154 raft_consensus.cc:2454] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d9f7441a9a4784ab803210fe1ce9d3 in term 0.
I20260807 03:08:38.801458 19013 raft_consensus.cc:2454] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7d9f7441a9a4784ab803210fe1ce9d3 in term 0.
I20260807 03:08:38.802232 18792 leader_election.cc:302] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 93f71254483b46f29e00456f62510c25, 9b60a62a8bc0455d89ca6d0870a41bd5, d7d9f7441a9a4784ab803210fe1ce9d3; no voters:
I20260807 03:08:38.802736 19230 raft_consensus.cc:2781] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:08:38.802896 19230 raft_consensus.cc:482] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:08:38.803021 19230 raft_consensus.cc:3037] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:38.805270 19230 raft_consensus.cc:504] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } }
I20260807 03:08:38.805907 19230 leader_election.cc:288] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [CANDIDATE]: Term 1 election: Requested vote from peers 93f71254483b46f29e00456f62510c25 (127.17.31.3:44737), 9b60a62a8bc0455d89ca6d0870a41bd5 (127.17.31.2:36411)
I20260807 03:08:38.806355 19013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91378dd62f254632bd3ce0c30499c2ba" candidate_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5"
I20260807 03:08:38.806363 19154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91378dd62f254632bd3ce0c30499c2ba" candidate_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f71254483b46f29e00456f62510c25"
I20260807 03:08:38.806569 19013 raft_consensus.cc:3037] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:38.806568 19154 raft_consensus.cc:3037] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:38.808693 19013 raft_consensus.cc:2454] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d9f7441a9a4784ab803210fe1ce9d3 in term 1.
I20260807 03:08:38.808697 19154 raft_consensus.cc:2454] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7d9f7441a9a4784ab803210fe1ce9d3 in term 1.
I20260807 03:08:38.809087 18792 leader_election.cc:302] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f71254483b46f29e00456f62510c25, d7d9f7441a9a4784ab803210fe1ce9d3; no voters:
I20260807 03:08:38.809389 19230 raft_consensus.cc:2781] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:08:38.809875 19230 raft_consensus.cc:686] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [term 1 LEADER]: Becoming Leader. State: Replica: d7d9f7441a9a4784ab803210fe1ce9d3, State: Running, Role: LEADER
I20260807 03:08:38.810468 19230 consensus_queue.cc:237] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } }
I20260807 03:08:38.815788 18699 catalog_manager.cc:5495] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7d9f7441a9a4784ab803210fe1ce9d3 (127.17.31.1). New cstate: current_term: 1 leader_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } health_report { overall_health: UNKNOWN } } }
I20260807 03:08:38.844893 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.839476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.17.31.1:45495)
0807 03:08:38.839653 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:38.839657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:38.839701 (+ 44us) server_negotiation.cc:407] Connection header received
0807 03:08:38.840006 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:38.840009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:38.840098 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:38.840220 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:08:38.840539 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.841252 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.842629 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.843101 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.843211 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.843754 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:08:38.844353 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:08:38.844472 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:38.844597 (+ 125us) server_negotiation.cc:299] Negotiation successful
0807 03:08:38.844668 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 03:08:38.853611 19013 raft_consensus.cc:1261] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 1 FOLLOWER]: Refusing update from remote peer d7d9f7441a9a4784ab803210fe1ce9d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:08:38.854638 19235 consensus_queue.cc:1035] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:08:38.855723 19154 raft_consensus.cc:1261] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25 [term 1 FOLLOWER]: Refusing update from remote peer d7d9f7441a9a4784ab803210fe1ce9d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:08:38.856536 19235 consensus_queue.cc:1035] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:08:38.867761 19239 mvcc.cc:205] Tried to move back new op lower bound from 7315751398813724672 to 7315751398652305408. Current Snapshot: MvccSnapshot[applied={T|T < 7315751398813724672}]
I20260807 03:08:38.867761 19240 mvcc.cc:205] Tried to move back new op lower bound from 7315751398813724672 to 7315751398652305408. Current Snapshot: MvccSnapshot[applied={T|T < 7315751398813724672}]
I20260807 03:08:38.886686 18699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:08:38.887055 18699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:08:38.891598 18699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960: Sending DeleteTablet for 3 replicas of tablet 91378dd62f254632bd3ce0c30499c2ba
I20260807 03:08:38.893222 19133 tablet_service.cc:1514] Processing DeleteTablet for tablet 91378dd62f254632bd3ce0c30499c2ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:38 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:48714
I20260807 03:08:38.893471 18840 tablet_service.cc:1514] Processing DeleteTablet for tablet 91378dd62f254632bd3ce0c30499c2ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:38 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:42400
I20260807 03:08:38.893647 18986 tablet_service.cc:1514] Processing DeleteTablet for tablet 91378dd62f254632bd3ce0c30499c2ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:38 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:35446
I20260807 03:08:38.898499 18699 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:08:38.899313 18699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:08:38.910041 19132 tablet_service.cc:1467] Processing CreateTablet for tablet 6a55d52ea4564f11a233e7162a63acf9 (DEFAULT_TABLE table=test-table [id=4d37e113bf5e43778115a5d1019e2f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:38.911780 18985 tablet_service.cc:1467] Processing CreateTablet for tablet 6a55d52ea4564f11a233e7162a63acf9 (DEFAULT_TABLE table=test-table [id=4d37e113bf5e43778115a5d1019e2f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:38.912528 19132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a55d52ea4564f11a233e7162a63acf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:38.912532 18985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a55d52ea4564f11a233e7162a63acf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:38.911720 18839 tablet_service.cc:1467] Processing CreateTablet for tablet 6a55d52ea4564f11a233e7162a63acf9 (DEFAULT_TABLE table=test-table [id=4d37e113bf5e43778115a5d1019e2f1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:38.914812 18839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a55d52ea4564f11a233e7162a63acf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:38.921926 19224 tablet_bootstrap.cc:492] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5: Bootstrap starting.
I20260807 03:08:38.923789 19224 tablet_bootstrap.cc:654] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:38.930027 19226 tablet_bootstrap.cc:492] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25: Bootstrap starting.
I20260807 03:08:38.931825 19226 tablet_bootstrap.cc:654] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:38.935516 19224 tablet_bootstrap.cc:492] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5: No bootstrap required, opened a new log
I20260807 03:08:38.935786 19224 ts_tablet_manager.cc:1397] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260807 03:08:38.936906 19224 raft_consensus.cc:348] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:38.937321 19224 raft_consensus.cc:374] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:38.937508 19224 raft_consensus.cc:729] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b60a62a8bc0455d89ca6d0870a41bd5, State: Initialized, Role: FOLLOWER
I20260807 03:08:38.937970 19224 consensus_queue.cc:260] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:38.939610 19225 tablet_bootstrap.cc:492] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3: Bootstrap starting.
I20260807 03:08:38.941362 19225 tablet_bootstrap.cc:654] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:38.941682 19224 ts_tablet_manager.cc:1428] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260807 03:08:38.945226 19226 tablet_bootstrap.cc:492] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25: No bootstrap required, opened a new log
I20260807 03:08:38.945505 19226 ts_tablet_manager.cc:1397] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260807 03:08:38.946573 19226 raft_consensus.cc:348] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:38.946933 19226 raft_consensus.cc:374] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:38.947080 19226 raft_consensus.cc:729] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f71254483b46f29e00456f62510c25, State: Initialized, Role: FOLLOWER
I20260807 03:08:38.947510 19226 consensus_queue.cc:260] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:38.947769 19249 tablet_replica.cc:330] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3: stopping tablet replica
I20260807 03:08:38.948652 19249 raft_consensus.cc:2229] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:08:38.949326 19248 tablet_replica.cc:330] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25: stopping tablet replica
I20260807 03:08:38.949445 19226 ts_tablet_manager.cc:1428] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260807 03:08:38.949781 19248 raft_consensus.cc:2229] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:08:38.950341 19248 raft_consensus.cc:2258] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:38.951432 19249 raft_consensus.cc:2258] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:38.952270 19225 tablet_bootstrap.cc:492] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3: No bootstrap required, opened a new log
I20260807 03:08:38.954473 19225 ts_tablet_manager.cc:1397] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.010s
I20260807 03:08:38.955399 19225 raft_consensus.cc:348] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:38.955804 19248 ts_tablet_manager.cc:1905] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:38.956336 19249 ts_tablet_manager.cc:1905] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:38.957069 19225 raft_consensus.cc:374] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:38.957265 19225 raft_consensus.cc:729] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d9f7441a9a4784ab803210fe1ce9d3, State: Initialized, Role: FOLLOWER
I20260807 03:08:38.957935 19250 tablet_replica.cc:330] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5: stopping tablet replica
I20260807 03:08:38.957787 19225 consensus_queue.cc:260] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:38.958364 19250 raft_consensus.cc:2229] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:08:38.958711 19225 ts_tablet_manager.cc:1428] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260807 03:08:38.958866 19250 raft_consensus.cc:2258] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:38.961198 19231 raft_consensus.cc:482] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:08:38.961361 19231 raft_consensus.cc:504] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:38.963066 19231 leader_election.cc:288] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7d9f7441a9a4784ab803210fe1ce9d3 (127.17.31.1:45495), 93f71254483b46f29e00456f62510c25 (127.17.31.3:44737)
I20260807 03:08:38.964339 19249 ts_tablet_manager.cc:1918] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:08:38.964622 19249 log.cc:1196] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/91378dd62f254632bd3ce0c30499c2ba
I20260807 03:08:38.965287 19249 ts_tablet_manager.cc:1939] T 91378dd62f254632bd3ce0c30499c2ba P d7d9f7441a9a4784ab803210fe1ce9d3: Deleting consensus metadata
I20260807 03:08:38.967097 18682 catalog_manager.cc:4841] TS d7d9f7441a9a4784ab803210fe1ce9d3 (127.17.31.1:45495): tablet 91378dd62f254632bd3ce0c30499c2ba (table test-table [id=ff47e5ce14de4ee6bd3c2965acfb2494]) successfully deleted
I20260807 03:08:38.971581 19250 ts_tablet_manager.cc:1905] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:38.982789 19248 ts_tablet_manager.cc:1918] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:08:38.983038 19248 log.cc:1196] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/91378dd62f254632bd3ce0c30499c2ba
I20260807 03:08:38.983660 19248 ts_tablet_manager.cc:1939] T 91378dd62f254632bd3ce0c30499c2ba P 93f71254483b46f29e00456f62510c25: Deleting consensus metadata
I20260807 03:08:38.985544 18685 catalog_manager.cc:4841] TS 93f71254483b46f29e00456f62510c25 (127.17.31.3:44737): tablet 91378dd62f254632bd3ce0c30499c2ba (table test-table [id=ff47e5ce14de4ee6bd3c2965acfb2494]) successfully deleted
I20260807 03:08:38.989137 19250 ts_tablet_manager.cc:1918] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:08:38.989256 19250 log.cc:1196] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/91378dd62f254632bd3ce0c30499c2ba
I20260807 03:08:38.989632 19250 ts_tablet_manager.cc:1939] T 91378dd62f254632bd3ce0c30499c2ba P 9b60a62a8bc0455d89ca6d0870a41bd5: Deleting consensus metadata
I20260807 03:08:38.990733 19254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.963520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45495 (local address 127.17.31.2:41803)
0807 03:08:38.978082 (+ 14562us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:38.978105 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:38.978826 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:38.979628 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:38.979633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:38.979658 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:38.980007 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.980017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.985646 (+ 5629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.985651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.988189 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.988203 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.989868 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.989878 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.990087 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.990121 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 03:08:38.990225 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 03:08:38.990378 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14442,"thread_start_us":14318,"threads_started":1}
I20260807 03:08:38.990777 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.963586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:41803 (local address 127.17.31.1:45495)
0807 03:08:38.963739 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:38.963745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:38.978190 (+ 14445us) server_negotiation.cc:407] Connection header received
0807 03:08:38.978967 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:38.978973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:38.979046 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:38.979103 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:38.982037 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.983179 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.988349 (+ 5170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.989618 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.989766 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.989886 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:38.990268 (+ 382us) server_negotiation.cc:299] Negotiation successful
0807 03:08:38.990448 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 03:08:38.991463 18682 catalog_manager.cc:4841] TS 9b60a62a8bc0455d89ca6d0870a41bd5 (127.17.31.2:36411): tablet 91378dd62f254632bd3ce0c30499c2ba (table test-table [id=ff47e5ce14de4ee6bd3c2965acfb2494]) successfully deleted
I20260807 03:08:38.992123 18864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a55d52ea4564f11a233e7162a63acf9" candidate_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" is_pre_election: true
I20260807 03:08:38.992584 18864 raft_consensus.cc:2454] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b60a62a8bc0455d89ca6d0870a41bd5 in term 0.
I20260807 03:08:38.993250 18935 leader_election.cc:302] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b60a62a8bc0455d89ca6d0870a41bd5, d7d9f7441a9a4784ab803210fe1ce9d3; no voters:
I20260807 03:08:38.993692 19231 raft_consensus.cc:2781] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:08:38.993737 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.964594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:55861 (local address 127.17.31.3:44737)
0807 03:08:38.964752 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:38.964758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:38.978611 (+ 13853us) server_negotiation.cc:407] Connection header received
0807 03:08:38.979313 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:38.979320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:38.979393 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:38.979452 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:38.980709 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.981871 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.991426 (+ 9555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.992644 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.992783 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.992884 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:38.993332 (+ 448us) server_negotiation.cc:299] Negotiation successful
0807 03:08:38.993482 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 03:08:38.993821 19255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:38.964269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:44737 (local address 127.17.31.2:55861)
0807 03:08:38.978525 (+ 14256us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:38.978540 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:38.978712 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:38.980187 (+ 1475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:38.980191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:38.980214 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:38.980525 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.980536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.983359 (+ 2823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.983364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.985467 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:38.985481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:38.992871 (+ 7390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:38.992878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:38.993120 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:38.993154 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 03:08:38.993316 (+ 162us) client_negotiation.cc:240] Negotiation successful
0807 03:08:38.993460 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14155,"thread_start_us":18038,"threads_started":1}
I20260807 03:08:38.994058 19231 raft_consensus.cc:482] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:08:38.994180 19231 raft_consensus.cc:3037] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:38.994539 19154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a55d52ea4564f11a233e7162a63acf9" candidate_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f71254483b46f29e00456f62510c25" is_pre_election: true
I20260807 03:08:38.994762 19154 raft_consensus.cc:2454] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b60a62a8bc0455d89ca6d0870a41bd5 in term 0.
I20260807 03:08:38.996439 19231 raft_consensus.cc:504] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:38.997021 19231 leader_election.cc:288] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [CANDIDATE]: Term 1 election: Requested vote from peers d7d9f7441a9a4784ab803210fe1ce9d3 (127.17.31.1:45495), 93f71254483b46f29e00456f62510c25 (127.17.31.3:44737)
I20260807 03:08:38.997430 18864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a55d52ea4564f11a233e7162a63acf9" candidate_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3"
I20260807 03:08:38.997506 19154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a55d52ea4564f11a233e7162a63acf9" candidate_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f71254483b46f29e00456f62510c25"
I20260807 03:08:38.997642 18864 raft_consensus.cc:3037] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:38.997705 19154 raft_consensus.cc:3037] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:38.999910 18864 raft_consensus.cc:2454] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b60a62a8bc0455d89ca6d0870a41bd5 in term 1.
I20260807 03:08:39.000399 18935 leader_election.cc:302] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b60a62a8bc0455d89ca6d0870a41bd5, d7d9f7441a9a4784ab803210fe1ce9d3; no voters:
I20260807 03:08:39.000656 19154 raft_consensus.cc:2454] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b60a62a8bc0455d89ca6d0870a41bd5 in term 1.
I20260807 03:08:39.000753 19231 raft_consensus.cc:2781] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:08:39.001446 19231 raft_consensus.cc:686] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 1 LEADER]: Becoming Leader. State: Replica: 9b60a62a8bc0455d89ca6d0870a41bd5, State: Running, Role: LEADER
I20260807 03:08:39.001892 19231 consensus_queue.cc:237] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:39.005148 18699 catalog_manager.cc:5495] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b60a62a8bc0455d89ca6d0870a41bd5 (127.17.31.2). New cstate: current_term: 1 leader_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } health_report { overall_health: UNKNOWN } } }
Aug 07 03:08:39 dist-test-slave-tm1g krb5kdc[18652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072116, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:39.062906 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:39.045105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44145 (local address 127.0.0.1:37912)
0807 03:08:39.046005 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:39.046027 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:39.046194 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:39.046614 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:39.046619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:39.047488 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:39.047834 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:39.047846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.049305 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.049311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:39.050921 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:39.050935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.051961 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.051967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:39.052146 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:39.052928 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:39.052955 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:39.055845 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:39.059730 (+ 3885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:39.059742 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:39.059749 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:39.060349 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:39.060762 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:39.060770 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:39.060774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:39.060949 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:39.061620 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:39.061629 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:39.061833 (+ 204us) client_negotiation.cc:769] Sending connection context
0807 03:08:39.062090 (+ 257us) client_negotiation.cc:240] Negotiation successful
0807 03:08:39.062441 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":570,"threads_started":1}
I20260807 03:08:39.063265 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:39.045068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37912 (local address 127.17.31.62:44145)
0807 03:08:39.045309 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:39.045316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:39.046136 (+ 820us) server_negotiation.cc:407] Connection header received
0807 03:08:39.046361 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:39.046367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:39.046446 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:39.046553 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:39.047990 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.049158 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.051081 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.051774 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.051902 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:39.056047 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:39.056053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:39.056058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:39.056091 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:39.059592 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:39.060481 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:39.060487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:39.060491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:39.060638 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:39.061073 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:39.061077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:39.061080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:39.061484 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:39.061623 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:39.062201 (+ 578us) server_negotiation.cc:299] Negotiation successful
0807 03:08:39.062617 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":170,"server-negotiator.queue_time_us":92}
I20260807 03:08:39.069569 18699 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:37912:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 03:08:39.071067 18699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:08:39.078104 18840 tablet_service.cc:1467] Processing CreateTablet for tablet 28691232133f44b3a8a9cf8b6b24afb3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4c46a77ca5847c79e85e53ddb365335]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 03:08:39.078094 18986 tablet_service.cc:1467] Processing CreateTablet for tablet 28691232133f44b3a8a9cf8b6b24afb3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4c46a77ca5847c79e85e53ddb365335]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 03:08:39.078337 19133 tablet_service.cc:1467] Processing CreateTablet for tablet 28691232133f44b3a8a9cf8b6b24afb3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4c46a77ca5847c79e85e53ddb365335]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 03:08:39.078727 18986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28691232133f44b3a8a9cf8b6b24afb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:39.078739 18840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28691232133f44b3a8a9cf8b6b24afb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:39.078841 19133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28691232133f44b3a8a9cf8b6b24afb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:39.086194 19225 tablet_bootstrap.cc:492] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3: Bootstrap starting.
I20260807 03:08:39.086294 19226 tablet_bootstrap.cc:492] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25: Bootstrap starting.
I20260807 03:08:39.087764 19224 tablet_bootstrap.cc:492] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5: Bootstrap starting.
I20260807 03:08:39.088004 19226 tablet_bootstrap.cc:654] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:39.088068 19225 tablet_bootstrap.cc:654] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:39.089607 19224 tablet_bootstrap.cc:654] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:39.098793 19226 tablet_bootstrap.cc:492] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25: No bootstrap required, opened a new log
I20260807 03:08:39.099057 19226 ts_tablet_manager.cc:1397] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260807 03:08:39.100076 19226 raft_consensus.cc:348] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:39.100512 19226 raft_consensus.cc:374] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:39.100666 19226 raft_consensus.cc:729] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f71254483b46f29e00456f62510c25, State: Initialized, Role: FOLLOWER
I20260807 03:08:39.101051 19226 consensus_queue.cc:260] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:39.101810 19226 ts_tablet_manager.cc:1428] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 03:08:39.101753 19232 tablet_replica.cc:439] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } }
I20260807 03:08:39.102093 19232 tablet_replica.cc:442] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:08:39.103533 19225 tablet_bootstrap.cc:492] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3: No bootstrap required, opened a new log
I20260807 03:08:39.103892 19225 ts_tablet_manager.cc:1397] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20260807 03:08:39.104241 19224 tablet_bootstrap.cc:492] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5: No bootstrap required, opened a new log
I20260807 03:08:39.104537 19224 ts_tablet_manager.cc:1397] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20260807 03:08:39.104753 19225 raft_consensus.cc:348] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:39.105094 19225 raft_consensus.cc:374] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:39.105229 19225 raft_consensus.cc:729] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d9f7441a9a4784ab803210fe1ce9d3, State: Initialized, Role: FOLLOWER
I20260807 03:08:39.105568 19225 consensus_queue.cc:260] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:39.105588 19224 raft_consensus.cc:348] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:39.105916 19224 raft_consensus.cc:374] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:39.106077 19224 raft_consensus.cc:729] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b60a62a8bc0455d89ca6d0870a41bd5, State: Initialized, Role: FOLLOWER
I20260807 03:08:39.106428 19224 consensus_queue.cc:260] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:39.106608 19225 ts_tablet_manager.cc:1428] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260807 03:08:39.106163 19243 tablet_replica.cc:439] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } }
I20260807 03:08:39.106956 19243 tablet_replica.cc:442] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:08:39.106904 19231 tablet_replica.cc:439] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } }
I20260807 03:08:39.107152 19231 tablet_replica.cc:442] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:08:39.108117 19224 ts_tablet_manager.cc:1428] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 03:08:39.140209 19231 raft_consensus.cc:482] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:08:39.140539 19231 raft_consensus.cc:504] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:39.141558 19231 leader_election.cc:288] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7d9f7441a9a4784ab803210fe1ce9d3 (127.17.31.1:45495), 93f71254483b46f29e00456f62510c25 (127.17.31.3:44737)
I20260807 03:08:39.142145 18864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28691232133f44b3a8a9cf8b6b24afb3" candidate_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" is_pre_election: true
I20260807 03:08:39.142179 19154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28691232133f44b3a8a9cf8b6b24afb3" candidate_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f71254483b46f29e00456f62510c25" is_pre_election: true
I20260807 03:08:39.142534 19154 raft_consensus.cc:2454] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b60a62a8bc0455d89ca6d0870a41bd5 in term 0.
I20260807 03:08:39.142622 18864 raft_consensus.cc:2454] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b60a62a8bc0455d89ca6d0870a41bd5 in term 0.
I20260807 03:08:39.143188 18937 leader_election.cc:302] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f71254483b46f29e00456f62510c25, 9b60a62a8bc0455d89ca6d0870a41bd5; no voters:
I20260807 03:08:39.143641 19231 raft_consensus.cc:2781] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:08:39.143783 19231 raft_consensus.cc:482] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:08:39.143926 19231 raft_consensus.cc:3037] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:39.146214 19231 raft_consensus.cc:504] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:39.146900 19231 leader_election.cc:288] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [CANDIDATE]: Term 1 election: Requested vote from peers d7d9f7441a9a4784ab803210fe1ce9d3 (127.17.31.1:45495), 93f71254483b46f29e00456f62510c25 (127.17.31.3:44737)
I20260807 03:08:39.147400 19154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28691232133f44b3a8a9cf8b6b24afb3" candidate_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93f71254483b46f29e00456f62510c25"
I20260807 03:08:39.147360 18864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28691232133f44b3a8a9cf8b6b24afb3" candidate_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3"
I20260807 03:08:39.147591 18864 raft_consensus.cc:3037] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:39.147655 19154 raft_consensus.cc:3037] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:39.150175 19154 raft_consensus.cc:2454] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b60a62a8bc0455d89ca6d0870a41bd5 in term 1.
I20260807 03:08:39.150624 18937 leader_election.cc:302] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93f71254483b46f29e00456f62510c25, 9b60a62a8bc0455d89ca6d0870a41bd5; no voters:
I20260807 03:08:39.151060 19231 raft_consensus.cc:2781] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:08:39.151335 19231 raft_consensus.cc:686] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 1 LEADER]: Becoming Leader. State: Replica: 9b60a62a8bc0455d89ca6d0870a41bd5, State: Running, Role: LEADER
I20260807 03:08:39.151671 19231 consensus_queue.cc:237] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } }
I20260807 03:08:39.152119 18864 raft_consensus.cc:2454] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b60a62a8bc0455d89ca6d0870a41bd5 in term 1.
I20260807 03:08:39.153918 19256 tablet_replica.cc:439] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5: TxnStatusTablet state changed. Reason: New leader 9b60a62a8bc0455d89ca6d0870a41bd5. Latest consensus state: current_term: 1 leader_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } }
I20260807 03:08:39.154646 19256 tablet_replica.cc:442] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5: This TxnStatusTablet replica's current role is: LEADER
I20260807 03:08:39.155529 18700 catalog_manager.cc:5495] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b60a62a8bc0455d89ca6d0870a41bd5 (127.17.31.2). New cstate: current_term: 1 leader_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } health_report { overall_health: UNKNOWN } } }
I20260807 03:08:39.156046 19269 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 03:08:39.156361 19269 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 03:08:39.186259 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:39.177432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:36411 (local address 127.0.0.1:35452)
0807 03:08:39.177688 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:39.177720 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:39.177865 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:39.178566 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:39.178571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:39.178585 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 03:08:39.178948 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:39.178961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.181100 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.181106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:39.183049 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:39.183065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.184198 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.184204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:39.184403 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:39.184517 (+ 114us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 03:08:39.185586 (+ 1069us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 03:08:39.185606 (+ 20us) client_negotiation.cc:769] Sending connection context
0807 03:08:39.185697 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 03:08:39.185873 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 03:08:39.186321 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:39.177564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.17.31.2:36411)
0807 03:08:39.177747 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:39.177753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:39.177778 (+ 25us) server_negotiation.cc:407] Connection header received
0807 03:08:39.178100 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:39.178106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:39.178295 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:39.178516 (+ 221us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:08:39.179118 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.180936 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.183200 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.184027 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.184148 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:39.184721 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:08:39.185447 (+ 726us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:08:39.185596 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:39.185730 (+ 134us) server_negotiation.cc:299] Negotiation successful
0807 03:08:39.185813 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:08:39.192490 19154 raft_consensus.cc:1261] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25 [term 1 FOLLOWER]: Refusing update from remote peer 9b60a62a8bc0455d89ca6d0870a41bd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:08:39.193020 18864 raft_consensus.cc:1261] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3 [term 1 FOLLOWER]: Refusing update from remote peer 9b60a62a8bc0455d89ca6d0870a41bd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:08:39.193651 19256 consensus_queue.cc:1035] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:08:39.194226 19231 consensus_queue.cc:1035] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:08:39.197937 19232 tablet_replica.cc:439] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25: TxnStatusTablet state changed. Reason: New leader 9b60a62a8bc0455d89ca6d0870a41bd5. Latest consensus state: current_term: 1 leader_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } }
I20260807 03:08:39.198292 19232 tablet_replica.cc:442] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:08:39.202147 19243 tablet_replica.cc:439] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3: TxnStatusTablet state changed. Reason: New leader 9b60a62a8bc0455d89ca6d0870a41bd5. Latest consensus state: current_term: 1 leader_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } }
I20260807 03:08:39.202570 19243 tablet_replica.cc:442] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:08:39.204495 19231 tablet_replica.cc:439] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } }
I20260807 03:08:39.204989 19231 tablet_replica.cc:442] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5: This TxnStatusTablet replica's current role is: LEADER
I20260807 03:08:39.207741 19256 tablet_replica.cc:439] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } }
I20260807 03:08:39.208118 19256 tablet_replica.cc:442] T 28691232133f44b3a8a9cf8b6b24afb3 P 9b60a62a8bc0455d89ca6d0870a41bd5: This TxnStatusTablet replica's current role is: LEADER
I20260807 03:08:39.209697 19232 tablet_replica.cc:439] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } }
I20260807 03:08:39.210070 19232 tablet_replica.cc:442] T 28691232133f44b3a8a9cf8b6b24afb3 P 93f71254483b46f29e00456f62510c25: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:08:39.214751 19238 mvcc.cc:205] Tried to move back new op lower bound from 7315751400202452992 to 7315751400047779840. Current Snapshot: MvccSnapshot[applied={T|T < 7315751400202452992}]
I20260807 03:08:39.215349 19243 tablet_replica.cc:439] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 } } peers { permanent_uuid: "9b60a62a8bc0455d89ca6d0870a41bd5" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 36411 } } peers { permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 } } }
I20260807 03:08:39.215631 19243 tablet_replica.cc:442] T 28691232133f44b3a8a9cf8b6b24afb3 P d7d9f7441a9a4784ab803210fe1ce9d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:08:39.228089 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:39.218752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35458 (local address 127.17.31.2:36411)
0807 03:08:39.218991 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:39.218998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:39.219032 (+ 34us) server_negotiation.cc:407] Connection header received
0807 03:08:39.219183 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:39.219189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:39.219306 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:39.219425 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:08:39.220232 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.221888 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.224806 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.226007 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.226137 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:39.226956 (+ 819us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:08:39.227117 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:08:39.227231 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:39.227468 (+ 237us) server_negotiation.cc:299] Negotiation successful
0807 03:08:39.227563 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260807 03:08:39.258358 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:39.246584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:53295 (local address 127.17.31.2:36411)
0807 03:08:39.246787 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:39.246793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:39.248074 (+ 1281us) server_negotiation.cc:407] Connection header received
0807 03:08:39.248351 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:39.248358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:39.248467 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:39.248595 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:08:39.249713 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.251509 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.254862 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.256228 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.256416 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:39.257135 (+ 719us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:08:39.257378 (+ 243us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:08:39.257501 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:39.257678 (+ 177us) server_negotiation.cc:299] Negotiation successful
0807 03:08:39.257767 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:08:39.258697 19276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:39.246745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:36411 (local address 127.17.31.2:53295)
0807 03:08:39.247933 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:39.247955 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:39.248179 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:39.248620 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:39.248626 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:39.248639 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 03:08:39.249491 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:39.249505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.251692 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.251712 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:39.254663 (+ 2951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:39.254678 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.256520 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.256527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:39.256787 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:39.256882 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 03:08:39.257501 (+ 619us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 03:08:39.257527 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 03:08:39.257639 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 03:08:39.257824 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"thread_start_us":728,"threads_started":1}
I20260807 03:08:39.275969 19154 raft_consensus.cc:1261] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25 [term 1 FOLLOWER]: Refusing update from remote peer 9b60a62a8bc0455d89ca6d0870a41bd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:08:39.276060 18864 raft_consensus.cc:1261] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3 [term 1 FOLLOWER]: Refusing update from remote peer 9b60a62a8bc0455d89ca6d0870a41bd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:08:39.276811 19277 consensus_queue.cc:1035] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93f71254483b46f29e00456f62510c25" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:08:39.277456 19256 consensus_queue.cc:1035] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7d9f7441a9a4784ab803210fe1ce9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:08:39.403496 18700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:08:39.403824 18700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:08:39.408082 18700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f7669eb1cb44212b2a0440c79c7e960: Sending DeleteTablet for 3 replicas of tablet 6a55d52ea4564f11a233e7162a63acf9
I20260807 03:08:39.409192 18840 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a55d52ea4564f11a233e7162a63acf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:39 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:42400
I20260807 03:08:39.409269 19133 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a55d52ea4564f11a233e7162a63acf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:39 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:48714
I20260807 03:08:39.409597 19248 tablet_replica.cc:330] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25: stopping tablet replica
I20260807 03:08:39.409617 19249 tablet_replica.cc:330] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3: stopping tablet replica
I20260807 03:08:39.409830 19248 raft_consensus.cc:2229] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:08:39.409873 19249 raft_consensus.cc:2229] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:08:39.410025 19248 raft_consensus.cc:2258] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:39.410110 19249 raft_consensus.cc:2258] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:39.411237 19248 ts_tablet_manager.cc:1905] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:39.412578 18986 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a55d52ea4564f11a233e7162a63acf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:39 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:35446
I20260807 03:08:39.412952 19250 tablet_replica.cc:330] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5: stopping tablet replica
I20260807 03:08:39.413225 19250 raft_consensus.cc:2229] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:08:39.413827 19250 raft_consensus.cc:2258] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:39.415488 19250 ts_tablet_manager.cc:1905] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:39.419852 19248 ts_tablet_manager.cc:1918] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 03:08:39.420120 19248 log.cc:1196] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/6a55d52ea4564f11a233e7162a63acf9
I20260807 03:08:39.420817 19248 ts_tablet_manager.cc:1939] T 6a55d52ea4564f11a233e7162a63acf9 P 93f71254483b46f29e00456f62510c25: Deleting consensus metadata
I20260807 03:08:39.422086 18685 catalog_manager.cc:4841] TS 93f71254483b46f29e00456f62510c25 (127.17.31.3:44737): tablet 6a55d52ea4564f11a233e7162a63acf9 (table test-table [id=4d37e113bf5e43778115a5d1019e2f1e]) successfully deleted
I20260807 03:08:39.424053 19249 ts_tablet_manager.cc:1905] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:39.430101 19249 ts_tablet_manager.cc:1918] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 03:08:39.430656 19249 log.cc:1196] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/6a55d52ea4564f11a233e7162a63acf9
I20260807 03:08:39.431398 19249 ts_tablet_manager.cc:1939] T 6a55d52ea4564f11a233e7162a63acf9 P d7d9f7441a9a4784ab803210fe1ce9d3: Deleting consensus metadata
I20260807 03:08:39.432612 18682 catalog_manager.cc:4841] TS d7d9f7441a9a4784ab803210fe1ce9d3 (127.17.31.1:45495): tablet 6a55d52ea4564f11a233e7162a63acf9 (table test-table [id=4d37e113bf5e43778115a5d1019e2f1e]) successfully deleted
I20260807 03:08:39.433243 19250 ts_tablet_manager.cc:1918] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 03:08:39.438603 19250 log.cc:1196] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/6a55d52ea4564f11a233e7162a63acf9
I20260807 03:08:39.439311 19250 ts_tablet_manager.cc:1939] T 6a55d52ea4564f11a233e7162a63acf9 P 9b60a62a8bc0455d89ca6d0870a41bd5: Deleting consensus metadata
I20260807 03:08:39.440809 18682 catalog_manager.cc:4841] TS 9b60a62a8bc0455d89ca6d0870a41bd5 (127.17.31.2:36411): tablet 6a55d52ea4564f11a233e7162a63acf9 (table test-table [id=4d37e113bf5e43778115a5d1019e2f1e]) successfully deleted
Aug 07 03:08:39 dist-test-slave-tm1g krb5kdc[18652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072118, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:08:39.459012 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:39.429485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42412 (local address 127.17.31.1:45495)
0807 03:08:39.429652 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:39.429658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:39.433717 (+ 4059us) server_negotiation.cc:407] Connection header received
0807 03:08:39.433982 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:39.433988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:39.434160 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:39.434262 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:39.437666 (+ 3404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.439162 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.441798 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.442667 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.442865 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:39.449496 (+ 6631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:39.449511 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:39.449536 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:39.449632 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:39.455011 (+ 5379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:39.456165 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:39.456180 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:39.456183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:39.456354 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:39.456874 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:39.456879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:39.456883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:39.457478 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:39.457839 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:39.458247 (+ 408us) server_negotiation.cc:299] Negotiation successful
0807 03:08:39.458541 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260807 03:08:39.460134 18909 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42412
I20260807 03:08:39.489588 18750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:39.466096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37926 (local address 127.17.31.62:44145)
0807 03:08:39.466441 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:39.466447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:39.467164 (+ 717us) server_negotiation.cc:407] Connection header received
0807 03:08:39.467503 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:39.467519 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:39.467666 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:39.467746 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:39.470600 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.472156 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.475248 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.476243 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.476391 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:39.480365 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:39.480381 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:39.480386 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:39.480448 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:39.485254 (+ 4806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:39.486419 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:39.486426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:39.486438 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:39.486696 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:39.487264 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:39.487270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:39.487274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:39.487809 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:39.488166 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:39.488734 (+ 568us) server_negotiation.cc:299] Negotiation successful
0807 03:08:39.489008 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
W20260807 03:08:39.490588 18700 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37926
I20260807 03:08:39.493286 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 18773
I20260807 03:08:39.517235 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:39.495212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.17.31.1:45495)
0807 03:08:39.495943 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:39.495950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:39.496738 (+ 788us) server_negotiation.cc:407] Connection header received
0807 03:08:39.496898 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:39.496911 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:39.497025 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:39.497174 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:39.498864 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.500674 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.502739 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.503742 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.503944 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:39.507041 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:39.507048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:39.507053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:39.507098 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:39.513016 (+ 5918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:39.513970 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:39.513985 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:39.513988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:39.514233 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:39.514790 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:39.514797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:39.514809 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:39.515365 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:39.515695 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:39.516006 (+ 311us) server_negotiation.cc:299] Negotiation successful
0807 03:08:39.516361 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487}
W20260807 03:08:39.518078 18909 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42420
W20260807 03:08:39.518836 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 18773: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:08:39.519058 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 18773
I20260807 03:08:39.548584 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 18918
Aug 07 03:08:39 dist-test-slave-tm1g krb5kdc[18652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072118, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:08:39.569211 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:39.549814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.17.31.2:36411)
0807 03:08:39.550048 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:39.550055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:39.550076 (+ 21us) server_negotiation.cc:407] Connection header received
0807 03:08:39.550142 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:39.550147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:39.550270 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:39.550406 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:39.552777 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.553989 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.555940 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.556617 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.556735 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:39.562020 (+ 5285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:39.562037 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:39.562056 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:39.562096 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:39.566108 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:39.566891 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:39.566908 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:39.566912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:39.567079 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:39.567507 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:39.567512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:39.567516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:39.567975 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:39.568307 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:39.568561 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 03:08:39.568824 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260807 03:08:39.570106 19056 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35462
W20260807 03:08:39.570775 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 18918: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:08:39.571035 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 18918
I20260807 03:08:39.592237 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 19066
Aug 07 03:08:39 dist-test-slave-tm1g krb5kdc[18652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072118, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:08:39.606523 19288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:39.592987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.17.31.3:44737)
0807 03:08:39.593510 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:39.593514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:39.593529 (+ 15us) server_negotiation.cc:407] Connection header received
0807 03:08:39.593578 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:39.593581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:39.593637 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:39.593724 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:39.594946 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.595692 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.596854 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:39.597250 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:39.597336 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:39.601344 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:39.601358 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:39.601373 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:39.601408 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:39.604213 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:39.604807 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:39.604813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:39.604815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:39.604954 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:39.605242 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:39.605245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:39.605247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:39.605578 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:39.605799 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:39.606021 (+ 222us) server_negotiation.cc:299] Negotiation successful
0807 03:08:39.606189 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":303,"threads_started":1}
W20260807 03:08:39.607149 19200 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48720
W20260807 03:08:39.607743 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 19066: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:08:39.607914 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 19066
I20260807 03:08:39.625952 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 18665
I20260807 03:08:39.627120 18741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:08:39.740790 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 18665
2026-08-07T03:08:39Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4491 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:08:39 dist-test-slave-tm1g krb5kdc[19292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:08:39 dist-test-slave-tm1g krb5kdc[19292](info): set up 2 sockets
Aug 07 03:08:39 dist-test-slave-tm1g krb5kdc[19292](info): commencing operation
krb5kdc: starting...
W20260807 03:08:40.848567 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:08:40 dist-test-slave-tm1g krb5kdc[19292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:08:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:08:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:08:41.031737 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:46185
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:46185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:41.346256 19305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:41.346632 19305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:41.346794 19305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:41.353709 19305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:08:41.353842 19305 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 03:08:41.353938 19305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:41.354051 19305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:41.354143 19305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:08:41.354228 19305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:08:41.364589 19305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:46185
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:46185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:41.365844 19305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:41.367247 19305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:41.378829 19310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:41.378891 19313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:41.378791 19311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:41.379851 19305 server_base.cc:1034] running on GCE node
I20260807 03:08:41.380570 19305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:41.381845 19305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:41.383086 19305 hybrid_clock.cc:648] HybridClock initialized: now 1786072121383024 us; error 38 us; skew 500 ppm
I20260807 03:08:41.383399 19305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:41 dist-test-slave-tm1g krb5kdc[19292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072121, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:41.392472 19305 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:08:41.393853 19305 webserver.cc:460] Webserver started at http://127.17.31.62:41397/ using document root <none> and password file <none>
I20260807 03:08:41.394641 19305 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:41.394773 19305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:41.395097 19305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:41.397910 19305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "7a2644f81bfa4181b1ec92e82b40ec97"
format_stamp: "Formatted at 2026-08-07 03:08:41 on dist-test-slave-tm1g"
server_key: "bc009bf1b8f0fd6c39016a7e5d52b858"
server_key_iv: "628f44dc03742b6de9f08b0d0d95c116"
server_key_version: "encryptionkey@0"
I20260807 03:08:41.398713 19305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "7a2644f81bfa4181b1ec92e82b40ec97"
format_stamp: "Formatted at 2026-08-07 03:08:41 on dist-test-slave-tm1g"
server_key: "bc009bf1b8f0fd6c39016a7e5d52b858"
server_key_iv: "628f44dc03742b6de9f08b0d0d95c116"
server_key_version: "encryptionkey@0"
I20260807 03:08:41.403589 19305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 03:08:41.406834 19320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:41.408136 19305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 03:08:41.408318 19305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "7a2644f81bfa4181b1ec92e82b40ec97"
format_stamp: "Formatted at 2026-08-07 03:08:41 on dist-test-slave-tm1g"
server_key: "bc009bf1b8f0fd6c39016a7e5d52b858"
server_key_iv: "628f44dc03742b6de9f08b0d0d95c116"
server_key_version: "encryptionkey@0"
I20260807 03:08:41.408538 19305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:41.458436 19305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:41.462417 19305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:41.462692 19305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:41.484050 19305 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:46185
I20260807 03:08:41.484083 19381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:46185 every 8 connection(s)
I20260807 03:08:41.485397 19305 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:08:41.488058 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 19305
I20260807 03:08:41.488266 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:08:41.489084 17532 external_mini_cluster.cc:1428] Setting key 962ab1db92dad746132b405477789272
I20260807 03:08:41.489590 19382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:41.503610 19382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97: Bootstrap starting.
Aug 07 03:08:41 dist-test-slave-tm1g krb5kdc[19292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:41.506891 19382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:41.507943 19382 log.cc:824] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97: Log is configured to *not* fsync() on all Append() calls
I20260807 03:08:41.510577 19382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97: No bootstrap required, opened a new log
I20260807 03:08:41.512072 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:41.491107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.17.31.62:46185)
0807 03:08:41.491891 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:41.491902 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:41.491940 (+ 38us) server_negotiation.cc:407] Connection header received
0807 03:08:41.494026 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:41.494046 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:41.494457 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:41.494777 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:41.496531 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:41.497769 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:41.499528 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:41.500236 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:41.500369 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:41.504995 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:41.505003 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:41.505018 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:41.505049 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:41.508091 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:41.508747 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:41.508754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:41.508759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:41.508929 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:41.509325 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:41.509329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:41.509331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:41.509781 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:41.510086 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:41.510604 (+ 518us) server_negotiation.cc:299] Negotiation successful
0807 03:08:41.510993 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":403,"threads_started":1}
I20260807 03:08:41.518100 19382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a2644f81bfa4181b1ec92e82b40ec97" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46185 } }
I20260807 03:08:41.518586 19382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:41.518675 19382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a2644f81bfa4181b1ec92e82b40ec97, State: Initialized, Role: FOLLOWER
I20260807 03:08:41.519261 19382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a2644f81bfa4181b1ec92e82b40ec97" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46185 } }
I20260807 03:08:41.519475 19382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:08:41.519611 19382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:08:41.519773 19382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:41.521739 19382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a2644f81bfa4181b1ec92e82b40ec97" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46185 } }
I20260807 03:08:41.522185 19382 leader_election.cc:302] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a2644f81bfa4181b1ec92e82b40ec97; no voters:
I20260807 03:08:41.522773 19382 leader_election.cc:288] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:08:41.522909 19387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:08:41.523656 19387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [term 1 LEADER]: Becoming Leader. State: Replica: 7a2644f81bfa4181b1ec92e82b40ec97, State: Running, Role: LEADER
I20260807 03:08:41.524250 19387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a2644f81bfa4181b1ec92e82b40ec97" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46185 } }
I20260807 03:08:41.524747 19382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:08:41.527058 19389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a2644f81bfa4181b1ec92e82b40ec97. Latest consensus state: current_term: 1 leader_uuid: "7a2644f81bfa4181b1ec92e82b40ec97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a2644f81bfa4181b1ec92e82b40ec97" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46185 } } }
I20260807 03:08:41.527428 19389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [sys.catalog]: This master's current role is: LEADER
I20260807 03:08:41.527907 19388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a2644f81bfa4181b1ec92e82b40ec97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a2644f81bfa4181b1ec92e82b40ec97" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46185 } } }
I20260807 03:08:41.528261 19388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97 [sys.catalog]: This master's current role is: LEADER
I20260807 03:08:41.531456 19396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:08:41.536149 19396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:08:41.544464 19396 catalog_manager.cc:1347] Generated new cluster ID: 1cc1059212d141a39674f5b441ccfd3a
I20260807 03:08:41.544579 19396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:08:41.557911 19396 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:08:41.559701 19396 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:08:41.567410 19396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97: Generated new TSK 0
I20260807 03:08:41.568331 19396 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:08:41.619237 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:46185
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:41.944276 19410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:41.944541 19410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:41.944707 19410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:41.951670 19410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:41.951764 19410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:41.951910 19410 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 03:08:41.951995 19410 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 03:08:41.952124 19410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:08:41.961300 19410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.31.62:46185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:41.962639 19410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:41.963932 19410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:41.977283 19416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:41.980455 19420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:41.980640 19417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:41.981396 19410 server_base.cc:1034] running on GCE node
I20260807 03:08:41.981961 19410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:41.982916 19410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:41.984148 19410 hybrid_clock.cc:648] HybridClock initialized: now 1786072121984099 us; error 40 us; skew 500 ppm
I20260807 03:08:41.984509 19410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:41 dist-test-slave-tm1g krb5kdc[19292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072121, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:41.988215 19410 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:08:41.989761 19410 webserver.cc:460] Webserver started at http://127.17.31.1:40227/ using document root <none> and password file <none>
I20260807 03:08:41.990638 19410 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:41.990763 19410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:41.991137 19410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:41.994185 19410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "19aed34633f6458f899019e9f54fcdc9"
format_stamp: "Formatted at 2026-08-07 03:08:41 on dist-test-slave-tm1g"
server_key: "e7d1079b79628690c1fd13bf5dff5b92"
server_key_iv: "40315424a4a01bad7fd666bb2def23db"
server_key_version: "encryptionkey@0"
I20260807 03:08:41.994987 19410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "19aed34633f6458f899019e9f54fcdc9"
format_stamp: "Formatted at 2026-08-07 03:08:41 on dist-test-slave-tm1g"
server_key: "e7d1079b79628690c1fd13bf5dff5b92"
server_key_iv: "40315424a4a01bad7fd666bb2def23db"
server_key_version: "encryptionkey@0"
I20260807 03:08:41.999722 19410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260807 03:08:42.002934 19428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:42.004200 19410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:08:42.004318 19410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "19aed34633f6458f899019e9f54fcdc9"
format_stamp: "Formatted at 2026-08-07 03:08:41 on dist-test-slave-tm1g"
server_key: "e7d1079b79628690c1fd13bf5dff5b92"
server_key_iv: "40315424a4a01bad7fd666bb2def23db"
server_key_version: "encryptionkey@0"
I20260807 03:08:42.004467 19410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:42.051786 19410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:42.055611 19410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:42.055914 19410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:42.058552 19410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:42.058674 19410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:42.058811 19410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:42.058910 19410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 03:08:42 dist-test-slave-tm1g krb5kdc[19292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072121, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:42.083958 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:42.059414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:33797 (local address 127.17.31.62:46185)
0807 03:08:42.059872 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:42.059879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:42.061167 (+ 1288us) server_negotiation.cc:407] Connection header received
0807 03:08:42.063641 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:42.063647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:42.063753 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:42.063877 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:42.066643 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:42.067914 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:42.070765 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:42.071316 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:42.071440 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:42.076978 (+ 5538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:42.076987 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:42.076993 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:42.077035 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:42.080368 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:42.081391 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:42.081398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:42.081402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:42.081569 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:42.082153 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:42.082158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:42.082162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:42.082495 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:42.082648 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:42.083347 (+ 699us) server_negotiation.cc:299] Negotiation successful
0807 03:08:42.083584 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":227,"threads_started":1}
I20260807 03:08:42.084980 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:42.060072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:46185 (local address 127.17.31.1:33797)
0807 03:08:42.060970 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:42.061026 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:42.063310 (+ 2284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:42.064211 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:42.064222 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:42.065237 (+ 1015us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:42.066343 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:42.066379 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:42.068128 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:42.068132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:42.070540 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:42.070563 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:42.071787 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:42.071798 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:42.072163 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:42.072978 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:42.073009 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:42.076699 (+ 3690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:42.080517 (+ 3818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:42.080526 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:42.080539 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:42.081196 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:42.081673 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:42.081678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:42.081682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:42.081971 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:42.082623 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:42.082631 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:42.082989 (+ 358us) client_negotiation.cc:769] Sending connection context
0807 03:08:42.083242 (+ 253us) client_negotiation.cc:240] Negotiation successful
0807 03:08:42.083731 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":414,"threads_started":1}
I20260807 03:08:42.108435 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:42.092012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:44817 (local address 127.17.31.62:46185)
0807 03:08:42.092176 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:42.092181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:42.092565 (+ 384us) server_negotiation.cc:407] Connection header received
0807 03:08:42.092745 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:42.092748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:42.092805 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:42.092879 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:42.094266 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:42.094990 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:42.097378 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:42.097747 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:42.097853 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:42.100580 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:42.100584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:42.100587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:42.100614 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:42.103704 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:42.105984 (+ 2280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:42.105989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:42.105993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:42.106158 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:42.106708 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:42.106715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:42.106722 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:42.107125 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:42.107290 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:42.107830 (+ 540us) server_negotiation.cc:299] Negotiation successful
0807 03:08:42.108076 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 03:08:42.108719 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:42.091738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:46185 (local address 127.17.31.1:44817)
0807 03:08:42.092439 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:42.092461 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:42.092624 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:42.092942 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:42.092950 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:42.093664 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:42.094105 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:42.094120 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:42.095122 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:42.095130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:42.097225 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:42.097244 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:42.097956 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:42.097966 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:42.098222 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:42.099042 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:42.099068 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:42.100395 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:42.103842 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:42.103847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:42.103851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:42.105785 (+ 1934us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:42.106292 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:42.106323 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:42.106335 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:42.106523 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:42.107370 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:42.107375 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:42.107557 (+ 182us) client_negotiation.cc:769] Sending connection context
0807 03:08:42.107705 (+ 148us) client_negotiation.cc:240] Negotiation successful
0807 03:08:42.108085 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":355,"threads_started":1}
I20260807 03:08:42.111393 19410 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:44901
I20260807 03:08:42.111477 19554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:44901 every 8 connection(s)
I20260807 03:08:42.112967 19410 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:08:42.114841 19555 heartbeater.cc:347] Connected to a master server at 127.17.31.62:46185
I20260807 03:08:42.115067 19555 heartbeater.cc:464] Registering TS with master...
I20260807 03:08:42.115649 19555 heartbeater.cc:511] Master 127.17.31.62:46185 requested a full tablet report, sending...
I20260807 03:08:42.116493 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 19410
I20260807 03:08:42.116699 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:08:42.117551 17532 external_mini_cluster.cc:1428] Setting key cdfb2db15348acbaebd7399577d571b8
I20260807 03:08:42.118175 19337 ts_manager.cc:194] Registered new tserver with Master: 19aed34633f6458f899019e9f54fcdc9 (127.17.31.1:44901)
I20260807 03:08:42.119818 19337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:33797
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:08:42.157827 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:46185
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:42.473284 19560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:42.473623 19560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:42.473803 19560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:42.480919 19560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:42.481065 19560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:42.481235 19560 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 03:08:42.481321 19560 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 03:08:42.481451 19560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:08:42.491263 19560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.31.62:46185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:42.492429 19560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:42.493702 19560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:42.506557 19566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:42.506525 19567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:42.506810 19570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:42.507299 19560 server_base.cc:1034] running on GCE node
I20260807 03:08:42.507951 19560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:42.508910 19560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:42.510129 19560 hybrid_clock.cc:648] HybridClock initialized: now 1786072122510084 us; error 58 us; skew 500 ppm
I20260807 03:08:42.510485 19560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:42 dist-test-slave-tm1g krb5kdc[19292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072122, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:42.513489 19560 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:08:42.514896 19560 webserver.cc:460] Webserver started at http://127.17.31.2:45201/ using document root <none> and password file <none>
I20260807 03:08:42.515563 19560 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:42.515709 19560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:42.516019 19560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:42.518863 19560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "6341165f95f14c2bb7df240e548bc06e"
format_stamp: "Formatted at 2026-08-07 03:08:42 on dist-test-slave-tm1g"
server_key: "ac0235949a6da2af6cb48c684f6db764"
server_key_iv: "dff903cf6ba47dad60938a2e1191fa62"
server_key_version: "encryptionkey@0"
I20260807 03:08:42.519609 19560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "6341165f95f14c2bb7df240e548bc06e"
format_stamp: "Formatted at 2026-08-07 03:08:42 on dist-test-slave-tm1g"
server_key: "ac0235949a6da2af6cb48c684f6db764"
server_key_iv: "dff903cf6ba47dad60938a2e1191fa62"
server_key_version: "encryptionkey@0"
I20260807 03:08:42.524428 19560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 03:08:42.527688 19577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:42.528842 19560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 03:08:42.529017 19560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "6341165f95f14c2bb7df240e548bc06e"
format_stamp: "Formatted at 2026-08-07 03:08:42 on dist-test-slave-tm1g"
server_key: "ac0235949a6da2af6cb48c684f6db764"
server_key_iv: "dff903cf6ba47dad60938a2e1191fa62"
server_key_version: "encryptionkey@0"
I20260807 03:08:42.529219 19560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:42.546546 19560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:42.550027 19560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:42.550345 19560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:42.552860 19560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:42.553001 19560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:42.553154 19560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:42.553260 19560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 03:08:42 dist-test-slave-tm1g krb5kdc[19292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072122, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:42.575974 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:42.553229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:40663 (local address 127.17.31.62:46185)
0807 03:08:42.553430 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:42.553440 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:42.554843 (+ 1403us) server_negotiation.cc:407] Connection header received
0807 03:08:42.556751 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:42.556756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:42.556877 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:42.556987 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:42.558963 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:42.560293 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:42.562761 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:42.563360 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:42.563489 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:42.569023 (+ 5534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:42.569031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:42.569036 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:42.569096 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:42.571981 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:42.572956 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:42.572964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:42.572971 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:42.573133 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:42.573694 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:42.573702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:42.573708 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:42.574150 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:42.574358 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:42.575121 (+ 763us) server_negotiation.cc:299] Negotiation successful
0807 03:08:42.575434 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 03:08:42.576373 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:42.553781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:46185 (local address 127.17.31.2:40663)
0807 03:08:42.554659 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:42.554712 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:42.556481 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:42.557206 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:42.557214 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:42.557902 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:42.558707 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:42.558725 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:42.560473 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:42.560480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:42.562598 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:42.562617 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:42.563609 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:42.563617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:42.564037 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:42.564832 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:42.564865 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:42.568745 (+ 3880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:42.572136 (+ 3391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:42.572148 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:42.572165 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:42.572817 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:42.573256 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:42.573261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:42.573265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:42.573534 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:42.574329 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:42.574336 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:42.574639 (+ 303us) client_negotiation.cc:769] Sending connection context
0807 03:08:42.574906 (+ 267us) client_negotiation.cc:240] Negotiation successful
0807 03:08:42.575302 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":368,"threads_started":1}
I20260807 03:08:42.597508 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:42.582952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:44351 (local address 127.17.31.62:46185)
0807 03:08:42.583167 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:42.583175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:42.583773 (+ 598us) server_negotiation.cc:407] Connection header received
0807 03:08:42.583980 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:42.583988 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:42.584096 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:42.584226 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:42.585374 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:42.586297 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:42.588802 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:42.589179 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:42.589268 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:42.591966 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:42.591971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:42.591974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:42.592003 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:42.594471 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:42.595308 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:42.595315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:42.595321 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:42.595486 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:42.595989 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:42.596003 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:42.596009 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:42.596376 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:42.596524 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:42.596962 (+ 438us) server_negotiation.cc:299] Negotiation successful
0807 03:08:42.597179 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 03:08:42.597633 19649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:42.582950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:46185 (local address 127.17.31.2:44351)
0807 03:08:42.583663 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:42.583685 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:42.583836 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:42.584233 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:42.584236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:42.584701 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:42.585203 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:42.585220 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:42.586463 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:42.586471 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:42.588649 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:42.588668 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:42.589400 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:42.589409 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:42.589678 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:42.590328 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:42.590358 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:42.591768 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:42.594610 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:42.594615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:42.594619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:42.595189 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:42.595635 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:42.595646 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:42.595652 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:42.595847 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:42.596520 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:42.596525 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:42.596706 (+ 181us) client_negotiation.cc:769] Sending connection context
0807 03:08:42.596865 (+ 159us) client_negotiation.cc:240] Negotiation successful
0807 03:08:42.597165 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":396,"threads_started":1}
I20260807 03:08:42.607945 19560 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:43991
I20260807 03:08:42.608017 19699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:43991 every 8 connection(s)
I20260807 03:08:42.609257 19560 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:08:42.611179 19700 heartbeater.cc:347] Connected to a master server at 127.17.31.62:46185
I20260807 03:08:42.611357 19700 heartbeater.cc:464] Registering TS with master...
I20260807 03:08:42.611922 19700 heartbeater.cc:511] Master 127.17.31.62:46185 requested a full tablet report, sending...
I20260807 03:08:42.613238 19337 ts_manager.cc:194] Registered new tserver with Master: 6341165f95f14c2bb7df240e548bc06e (127.17.31.2:43991)
I20260807 03:08:42.613437 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 19560
I20260807 03:08:42.613619 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:08:42.614187 17532 external_mini_cluster.cc:1428] Setting key 86281fbeb0478885469ea64265479d4e
I20260807 03:08:42.614416 19337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:40663
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:08:42.655138 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:46185
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:42.972278 19704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:42.972540 19704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:42.972638 19704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:42.979962 19704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:42.980057 19704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:42.980139 19704 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 03:08:42.980189 19704 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 03:08:42.980257 19704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:08:42.989741 19704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.31.62:46185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:42.991014 19704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:42.992250 19704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:43.004885 19712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:43.004948 19709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:43.004981 19710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:43.010237 19704 server_base.cc:1034] running on GCE node
I20260807 03:08:43.010867 19704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:43.011782 19704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:43.012998 19704 hybrid_clock.cc:648] HybridClock initialized: now 1786072123012959 us; error 36 us; skew 500 ppm
I20260807 03:08:43.013298 19704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:43 dist-test-slave-tm1g krb5kdc[19292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072123, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:43.016342 19704 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:08:43.017765 19704 webserver.cc:460] Webserver started at http://127.17.31.3:39889/ using document root <none> and password file <none>
I20260807 03:08:43.018584 19704 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:43.018733 19704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:43.019003 19704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:43.021709 19704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "81fb0785479c4ffe99953381648d2e97"
format_stamp: "Formatted at 2026-08-07 03:08:43 on dist-test-slave-tm1g"
server_key: "4a450329fd32046ab10f40f6614ab368"
server_key_iv: "5466a2d0cf7cdb1640edca07fa530f69"
server_key_version: "encryptionkey@0"
I20260807 03:08:43.022470 19704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "81fb0785479c4ffe99953381648d2e97"
format_stamp: "Formatted at 2026-08-07 03:08:43 on dist-test-slave-tm1g"
server_key: "4a450329fd32046ab10f40f6614ab368"
server_key_iv: "5466a2d0cf7cdb1640edca07fa530f69"
server_key_version: "encryptionkey@0"
I20260807 03:08:43.027491 19704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 03:08:43.030712 19719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:43.031940 19704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 03:08:43.032066 19704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "81fb0785479c4ffe99953381648d2e97"
format_stamp: "Formatted at 2026-08-07 03:08:43 on dist-test-slave-tm1g"
server_key: "4a450329fd32046ab10f40f6614ab368"
server_key_iv: "5466a2d0cf7cdb1640edca07fa530f69"
server_key_version: "encryptionkey@0"
I20260807 03:08:43.032261 19704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:43.047065 19704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:43.050724 19704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:43.051005 19704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:43.053539 19704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:43.053668 19704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:43.053856 19704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:43.053951 19704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 03:08:43 dist-test-slave-tm1g krb5kdc[19292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072123, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:43.075461 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.053910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:50405 (local address 127.17.31.62:46185)
0807 03:08:43.054124 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:43.054133 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:43.055588 (+ 1455us) server_negotiation.cc:407] Connection header received
0807 03:08:43.057548 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:43.057554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:43.057656 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:43.057794 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:43.060020 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.061109 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.063753 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.064134 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.064270 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.069244 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:43.069249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:43.069272 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:43.069324 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:43.072369 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:43.073082 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:43.073087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:43.073091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:43.073232 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:43.073675 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:43.073679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:43.073683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:43.073951 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:43.074122 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:43.074864 (+ 742us) server_negotiation.cc:299] Negotiation successful
0807 03:08:43.075093 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 03:08:43.076423 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.054571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:46185 (local address 127.17.31.3:50405)
0807 03:08:43.055404 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:43.055451 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:43.057277 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:43.058104 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:43.058114 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:43.058910 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:43.059768 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.059797 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.061270 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.061275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.063545 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.063563 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.064388 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.064396 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.064896 (+ 500us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.065638 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:43.065676 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:43.068972 (+ 3296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:43.072503 (+ 3531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:43.072514 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:43.072528 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:43.072945 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:43.073345 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:43.073350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:43.073354 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:43.073549 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:43.074081 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:43.074089 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:43.074493 (+ 404us) client_negotiation.cc:769] Sending connection context
0807 03:08:43.074757 (+ 264us) client_negotiation.cc:240] Negotiation successful
0807 03:08:43.075276 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":376,"threads_started":1}
I20260807 03:08:43.098699 19788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.083383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:46185 (local address 127.17.31.3:48875)
0807 03:08:43.084067 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:43.084082 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:43.084181 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:43.084532 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:43.084535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:43.084919 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:43.085195 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.085203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.086894 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.086902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.088353 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.088365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.089604 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.089608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.089732 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.090425 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:43.090440 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:43.091616 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:43.095901 (+ 4285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:43.095907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:43.095911 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:43.096538 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:43.096928 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:43.096933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:43.096936 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:43.097061 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:43.097553 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:43.097558 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:43.097736 (+ 178us) client_negotiation.cc:769] Sending connection context
0807 03:08:43.097886 (+ 150us) client_negotiation.cc:240] Negotiation successful
0807 03:08:43.098210 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":387,"threads_started":1}
I20260807 03:08:43.098834 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.083388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:48875 (local address 127.17.31.62:46185)
0807 03:08:43.083592 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:43.083600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:43.084163 (+ 563us) server_negotiation.cc:407] Connection header received
0807 03:08:43.084293 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:43.084301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:43.084395 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:43.084522 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:43.085337 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.086716 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.088496 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.089426 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.089573 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.091916 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:43.091926 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:43.091933 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:43.091973 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:43.095792 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:43.096664 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:43.096671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:43.096677 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:43.096810 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:43.097176 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:43.097181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:43.097184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:43.097430 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:43.097597 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:43.098242 (+ 645us) server_negotiation.cc:299] Negotiation successful
0807 03:08:43.098484 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 03:08:43.113258 19704 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:39065
I20260807 03:08:43.113354 19845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:39065 every 8 connection(s)
I20260807 03:08:43.114511 19704 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:08:43.116595 19846 heartbeater.cc:347] Connected to a master server at 127.17.31.62:46185
I20260807 03:08:43.116783 19846 heartbeater.cc:464] Registering TS with master...
I20260807 03:08:43.117340 19846 heartbeater.cc:511] Master 127.17.31.62:46185 requested a full tablet report, sending...
I20260807 03:08:43.118618 19337 ts_manager.cc:194] Registered new tserver with Master: 81fb0785479c4ffe99953381648d2e97 (127.17.31.3:39065)
I20260807 03:08:43.119376 19337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:50405
I20260807 03:08:43.121714 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 19704
I20260807 03:08:43.121858 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:08:43.122371 17532 external_mini_cluster.cc:1428] Setting key 606f2903d7182e409b256adc4b609942
I20260807 03:08:43.122334 19555 heartbeater.cc:503] Master 127.17.31.62:46185 was elected leader, sending a full tablet report...
I20260807 03:08:43.125335 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 03:08:43 dist-test-slave-tm1g krb5kdc[19292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 03:08:43 dist-test-slave-tm1g krb5kdc[19292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:43.173133 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.160189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44612 (local address 127.17.31.62:46185)
0807 03:08:43.160435 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:43.160442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:43.160571 (+ 129us) server_negotiation.cc:407] Connection header received
0807 03:08:43.160721 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:43.160725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:43.160792 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:43.160879 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:43.162248 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.163068 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.164241 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.164638 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.164728 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.168449 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:43.168453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:43.168456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:43.168487 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:43.170936 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:43.171508 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:43.171512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:43.171514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:43.171628 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:43.171934 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:43.171937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:43.171938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:43.172197 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:43.172375 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:43.172611 (+ 236us) server_negotiation.cc:299] Negotiation successful
0807 03:08:43.172765 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 03:08:43.177042 19337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:08:43.184126 19337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:08:43.216763 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.196317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.17.31.3:39065)
0807 03:08:43.207839 (+ 11522us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:43.207848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:43.208329 (+ 481us) server_negotiation.cc:407] Connection header received
0807 03:08:43.208575 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:43.208583 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:43.208851 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:43.209054 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:43.210005 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.211617 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.214295 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.215243 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.215409 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.215650 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:43.216219 (+ 569us) server_negotiation.cc:299] Negotiation successful
0807 03:08:43.216388 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11361,"thread_start_us":11183,"threads_started":1}
I20260807 03:08:43.217387 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.195928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:39065 (local address 127.0.0.1:50320)
0807 03:08:43.207887 (+ 11959us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:43.207934 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:43.208144 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:43.209092 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:43.209104 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:43.209173 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:43.209757 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.209771 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.211785 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.211790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.214140 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.214151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.215769 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.215778 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.215970 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.216016 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 03:08:43.216150 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 03:08:43.216340 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11059,"mutex_wait_us":84,"thread_start_us":13404,"threads_started":1}
I20260807 03:08:43.218250 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.196047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:43991 (local address 127.0.0.1:34206)
0807 03:08:43.208707 (+ 12660us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:43.208725 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:43.208845 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:43.210580 (+ 1735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:43.210584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:43.210609 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:43.211115 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.211125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.212967 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.212972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.215389 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.215399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.217420 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.217426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.217645 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.217675 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 03:08:43.217754 (+ 79us) client_negotiation.cc:240] Negotiation successful
0807 03:08:43.217886 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12557,"mutex_wait_us":151,"thread_start_us":10787,"threads_started":1}
I20260807 03:08:43.218556 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.198926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37482 (local address 127.17.31.1:44901)
0807 03:08:43.208294 (+ 9368us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:43.208301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:43.209423 (+ 1122us) server_negotiation.cc:407] Connection header received
0807 03:08:43.209714 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:43.209724 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:43.209952 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:43.210169 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:43.211125 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.212351 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.215354 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.216793 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.216934 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.217244 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:43.218078 (+ 834us) server_negotiation.cc:299] Negotiation successful
0807 03:08:43.218237 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9176,"thread_start_us":9008,"threads_started":1}
I20260807 03:08:43.219233 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.196406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.17.31.2:43991)
0807 03:08:43.209735 (+ 13329us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:43.209744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:43.209770 (+ 26us) server_negotiation.cc:407] Connection header received
0807 03:08:43.209878 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:43.209888 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:43.210122 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:43.210576 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:43.211298 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.212776 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.215528 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.217182 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.217312 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.217510 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:43.218867 (+ 1357us) server_negotiation.cc:299] Negotiation successful
0807 03:08:43.219025 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13151,"thread_start_us":13878,"threads_started":1}
I20260807 03:08:43.221105 19772 tablet_service.cc:1467] Processing CreateTablet for tablet be1cb2f50f514bdda6cb8dc1179446ff (DEFAULT_TABLE table=test-table [id=8a387503cde44e97b0a8a028dd28e924]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:43.222502 19772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1cb2f50f514bdda6cb8dc1179446ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:43.222084 19628 tablet_service.cc:1467] Processing CreateTablet for tablet be1cb2f50f514bdda6cb8dc1179446ff (DEFAULT_TABLE table=test-table [id=8a387503cde44e97b0a8a028dd28e924]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:43.223172 19628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1cb2f50f514bdda6cb8dc1179446ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:43.225073 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.198183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:44901 (local address 127.0.0.1:37482)
0807 03:08:43.209312 (+ 11129us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:43.209330 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:43.209529 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:43.210441 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:43.210444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:43.210463 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:43.210938 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.210952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.212550 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.212555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.215184 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.215198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.217448 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.217457 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.217886 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.217936 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 03:08:43.218025 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 03:08:43.218147 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11020,"mutex_wait_us":165,"thread_start_us":10270,"threads_started":1}
I20260807 03:08:43.227973 19481 tablet_service.cc:1467] Processing CreateTablet for tablet be1cb2f50f514bdda6cb8dc1179446ff (DEFAULT_TABLE table=test-table [id=8a387503cde44e97b0a8a028dd28e924]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:43.228973 19481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1cb2f50f514bdda6cb8dc1179446ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:43.235239 19865 tablet_bootstrap.cc:492] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97: Bootstrap starting.
I20260807 03:08:43.238972 19865 tablet_bootstrap.cc:654] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:43.240265 19865 log.cc:824] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97: Log is configured to *not* fsync() on all Append() calls
I20260807 03:08:43.241477 19867 tablet_bootstrap.cc:492] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9: Bootstrap starting.
I20260807 03:08:43.243081 19866 tablet_bootstrap.cc:492] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e: Bootstrap starting.
I20260807 03:08:43.245105 19867 tablet_bootstrap.cc:654] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:43.246021 19867 log.cc:824] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9: Log is configured to *not* fsync() on all Append() calls
I20260807 03:08:43.246352 19865 tablet_bootstrap.cc:492] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97: No bootstrap required, opened a new log
I20260807 03:08:43.246383 19866 tablet_bootstrap.cc:654] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:43.246621 19865 ts_tablet_manager.cc:1397] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260807 03:08:43.247568 19866 log.cc:824] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e: Log is configured to *not* fsync() on all Append() calls
I20260807 03:08:43.248942 19867 tablet_bootstrap.cc:492] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9: No bootstrap required, opened a new log
I20260807 03:08:43.249189 19867 ts_tablet_manager.cc:1397] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 03:08:43.250378 19866 tablet_bootstrap.cc:492] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e: No bootstrap required, opened a new log
I20260807 03:08:43.250631 19866 ts_tablet_manager.cc:1397] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260807 03:08:43.255725 19867 raft_consensus.cc:348] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } }
I20260807 03:08:43.256057 19867 raft_consensus.cc:374] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:43.256177 19867 raft_consensus.cc:729] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19aed34633f6458f899019e9f54fcdc9, State: Initialized, Role: FOLLOWER
I20260807 03:08:43.256834 19867 consensus_queue.cc:260] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } }
I20260807 03:08:43.256929 19865 raft_consensus.cc:348] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } }
I20260807 03:08:43.257423 19865 raft_consensus.cc:374] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:43.257558 19865 raft_consensus.cc:729] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81fb0785479c4ffe99953381648d2e97, State: Initialized, Role: FOLLOWER
I20260807 03:08:43.258240 19865 consensus_queue.cc:260] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } }
I20260807 03:08:43.259861 19867 ts_tablet_manager.cc:1428] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
I20260807 03:08:43.260767 19846 heartbeater.cc:503] Master 127.17.31.62:46185 was elected leader, sending a full tablet report...
I20260807 03:08:43.261384 19865 ts_tablet_manager.cc:1428] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260807 03:08:43.261226 19866 raft_consensus.cc:348] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } }
I20260807 03:08:43.261615 19866 raft_consensus.cc:374] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:43.261754 19866 raft_consensus.cc:729] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6341165f95f14c2bb7df240e548bc06e, State: Initialized, Role: FOLLOWER
I20260807 03:08:43.263113 19866 consensus_queue.cc:260] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } }
I20260807 03:08:43.264137 19700 heartbeater.cc:503] Master 127.17.31.62:46185 was elected leader, sending a full tablet report...
I20260807 03:08:43.264720 19866 ts_tablet_manager.cc:1428] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260807 03:08:43.353384 19873 raft_consensus.cc:482] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:08:43.353678 19873 raft_consensus.cc:504] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } }
I20260807 03:08:43.355499 19873 leader_election.cc:288] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81fb0785479c4ffe99953381648d2e97 (127.17.31.3:39065), 6341165f95f14c2bb7df240e548bc06e (127.17.31.2:43991)
W20260807 03:08:43.361366 19701 tablet.cc:2369] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:08:43.363381 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.355838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:52847 (local address 127.17.31.3:39065)
0807 03:08:43.356055 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:43.356062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:43.356801 (+ 739us) server_negotiation.cc:407] Connection header received
0807 03:08:43.357014 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:43.357022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:43.357105 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:43.357194 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:43.357843 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.358731 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.361290 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.362515 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.362644 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.362753 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:43.362997 (+ 244us) server_negotiation.cc:299] Negotiation successful
0807 03:08:43.363122 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 03:08:43.363503 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.355693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:39065 (local address 127.17.31.1:52847)
0807 03:08:43.356682 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:43.356701 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:43.356869 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:43.357240 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:43.357245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:43.357271 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:43.357684 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.357697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.358856 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.358861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.361064 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.361079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.362695 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.362700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.362812 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.362837 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 03:08:43.362960 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 03:08:43.363084 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":113,"thread_start_us":584,"threads_started":1}
I20260807 03:08:43.363591 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.356076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:44627 (local address 127.17.31.2:43991)
0807 03:08:43.356300 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:43.356306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:43.357088 (+ 782us) server_negotiation.cc:407] Connection header received
0807 03:08:43.357260 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:43.357265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:43.357342 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:43.357444 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:43.358144 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.359010 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.361185 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.362516 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.362644 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.362753 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:43.363210 (+ 457us) server_negotiation.cc:299] Negotiation successful
0807 03:08:43.363334 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 03:08:43.363766 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.356099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:43991 (local address 127.17.31.1:44627)
0807 03:08:43.356959 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:43.356981 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:43.357116 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:43.357524 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:43.357532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:43.357561 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:43.357982 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.357993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.359174 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.359179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.361026 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.361037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.362935 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.362939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.363057 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.363079 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 03:08:43.363183 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 03:08:43.363308 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":5,"thread_start_us":471,"threads_started":1}
I20260807 03:08:43.364560 19799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1cb2f50f514bdda6cb8dc1179446ff" candidate_uuid: "19aed34633f6458f899019e9f54fcdc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81fb0785479c4ffe99953381648d2e97" is_pre_election: true
I20260807 03:08:43.364781 19652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1cb2f50f514bdda6cb8dc1179446ff" candidate_uuid: "19aed34633f6458f899019e9f54fcdc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6341165f95f14c2bb7df240e548bc06e" is_pre_election: true
I20260807 03:08:43.365065 19799 raft_consensus.cc:2454] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19aed34633f6458f899019e9f54fcdc9 in term 0.
W20260807 03:08:43.365305 19556 tablet.cc:2369] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:08:43.365319 19652 raft_consensus.cc:2454] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19aed34633f6458f899019e9f54fcdc9 in term 0.
I20260807 03:08:43.365815 19429 leader_election.cc:302] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19aed34633f6458f899019e9f54fcdc9, 81fb0785479c4ffe99953381648d2e97; no voters:
I20260807 03:08:43.366263 19873 raft_consensus.cc:2781] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:08:43.366425 19873 raft_consensus.cc:482] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:08:43.366515 19873 raft_consensus.cc:3037] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:08:43.366555 19847 tablet.cc:2369] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:08:43.368736 19873 raft_consensus.cc:504] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } }
I20260807 03:08:43.369313 19873 leader_election.cc:288] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [CANDIDATE]: Term 1 election: Requested vote from peers 81fb0785479c4ffe99953381648d2e97 (127.17.31.3:39065), 6341165f95f14c2bb7df240e548bc06e (127.17.31.2:43991)
I20260807 03:08:43.369745 19652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1cb2f50f514bdda6cb8dc1179446ff" candidate_uuid: "19aed34633f6458f899019e9f54fcdc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6341165f95f14c2bb7df240e548bc06e"
I20260807 03:08:43.369750 19799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1cb2f50f514bdda6cb8dc1179446ff" candidate_uuid: "19aed34633f6458f899019e9f54fcdc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81fb0785479c4ffe99953381648d2e97"
I20260807 03:08:43.369933 19652 raft_consensus.cc:3037] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:43.369942 19799 raft_consensus.cc:3037] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:43.372146 19652 raft_consensus.cc:2454] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19aed34633f6458f899019e9f54fcdc9 in term 1.
I20260807 03:08:43.372210 19799 raft_consensus.cc:2454] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19aed34633f6458f899019e9f54fcdc9 in term 1.
I20260807 03:08:43.372568 19430 leader_election.cc:302] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19aed34633f6458f899019e9f54fcdc9, 6341165f95f14c2bb7df240e548bc06e; no voters:
I20260807 03:08:43.372897 19873 raft_consensus.cc:2781] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:08:43.373301 19873 raft_consensus.cc:686] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [term 1 LEADER]: Becoming Leader. State: Replica: 19aed34633f6458f899019e9f54fcdc9, State: Running, Role: LEADER
I20260807 03:08:43.373754 19873 consensus_queue.cc:237] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } }
I20260807 03:08:43.378264 19337 catalog_manager.cc:5495] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19aed34633f6458f899019e9f54fcdc9 (127.17.31.1). New cstate: current_term: 1 leader_uuid: "19aed34633f6458f899019e9f54fcdc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } health_report { overall_health: HEALTHY } } }
Aug 07 03:08:43 dist-test-slave-tm1g krb5kdc[19292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072121, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:43.437656 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.424711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:46185 (local address 127.0.0.1:44614)
0807 03:08:43.425255 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:43.425270 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:43.425440 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:43.425806 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:43.425810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:43.426571 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:43.426841 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.426850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.427843 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.427851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.428800 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.428811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.429556 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.429561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.429792 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.430431 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:43.430448 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:43.432430 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:43.435383 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:43.435392 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:43.435397 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:43.435852 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:43.436170 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:43.436173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:43.436175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:43.436362 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:43.436864 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:43.436869 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:43.437007 (+ 138us) client_negotiation.cc:769] Sending connection context
0807 03:08:43.437137 (+ 130us) client_negotiation.cc:240] Negotiation successful
0807 03:08:43.437327 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":290,"threads_started":1}
I20260807 03:08:43.437695 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.424652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.17.31.62:46185)
0807 03:08:43.424824 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:43.424829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:43.425366 (+ 537us) server_negotiation.cc:407] Connection header received
0807 03:08:43.425555 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:43.425560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:43.425659 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:43.425760 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:43.426974 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.427724 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.428939 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.429382 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.429524 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.432614 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:43.432618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:43.432622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:43.432649 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:43.435278 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:43.435953 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:43.435956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:43.435959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:43.436067 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:43.436454 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:43.436457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:43.436459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:43.436746 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:43.436867 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:43.437218 (+ 351us) server_negotiation.cc:299] Negotiation successful
0807 03:08:43.437447 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":59}
I20260807 03:08:43.442368 19337 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:44614:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 03:08:43.443311 19337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:08:43.449625 19628 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc24dde9fb249ee8e88c5491c28b141 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fc3382f1c7b14150b63a1b7cc0e59fe8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 03:08:43.449409 19481 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc24dde9fb249ee8e88c5491c28b141 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fc3382f1c7b14150b63a1b7cc0e59fe8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 03:08:43.449674 19772 tablet_service.cc:1467] Processing CreateTablet for tablet 0bc24dde9fb249ee8e88c5491c28b141 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fc3382f1c7b14150b63a1b7cc0e59fe8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 03:08:43.450438 19772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc24dde9fb249ee8e88c5491c28b141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:43.450440 19481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc24dde9fb249ee8e88c5491c28b141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:43.450464 19628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bc24dde9fb249ee8e88c5491c28b141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:43.455864 19866 tablet_bootstrap.cc:492] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e: Bootstrap starting.
I20260807 03:08:43.457485 19867 tablet_bootstrap.cc:492] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9: Bootstrap starting.
I20260807 03:08:43.457671 19866 tablet_bootstrap.cc:654] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:43.458511 19865 tablet_bootstrap.cc:492] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97: Bootstrap starting.
I20260807 03:08:43.459261 19867 tablet_bootstrap.cc:654] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:43.459991 19865 tablet_bootstrap.cc:654] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:43.460078 19866 tablet_bootstrap.cc:492] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e: No bootstrap required, opened a new log
I20260807 03:08:43.460259 19866 ts_tablet_manager.cc:1397] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260807 03:08:43.461247 19866 raft_consensus.cc:348] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:43.461722 19866 raft_consensus.cc:374] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:43.461841 19866 raft_consensus.cc:729] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6341165f95f14c2bb7df240e548bc06e, State: Initialized, Role: FOLLOWER
I20260807 03:08:43.462114 19865 tablet_bootstrap.cc:492] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97: No bootstrap required, opened a new log
I20260807 03:08:43.462256 19865 ts_tablet_manager.cc:1397] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 03:08:43.462141 19866 consensus_queue.cc:260] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:43.462409 19867 tablet_bootstrap.cc:492] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9: No bootstrap required, opened a new log
I20260807 03:08:43.462548 19867 ts_tablet_manager.cc:1397] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260807 03:08:43.462877 19866 ts_tablet_manager.cc:1428] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 03:08:43.462672 19875 tablet_replica.cc:439] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } }
I20260807 03:08:43.463095 19875 tablet_replica.cc:442] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:08:43.463093 19865 raft_consensus.cc:348] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:43.463361 19865 raft_consensus.cc:374] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:43.463459 19865 raft_consensus.cc:729] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81fb0785479c4ffe99953381648d2e97, State: Initialized, Role: FOLLOWER
I20260807 03:08:43.463327 19867 raft_consensus.cc:348] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:43.463580 19867 raft_consensus.cc:374] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:43.463745 19867 raft_consensus.cc:729] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19aed34633f6458f899019e9f54fcdc9, State: Initialized, Role: FOLLOWER
I20260807 03:08:43.463709 19865 consensus_queue.cc:260] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:43.464017 19867 consensus_queue.cc:260] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:43.464304 19865 ts_tablet_manager.cc:1428] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 03:08:43.464187 19874 tablet_replica.cc:439] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } }
I20260807 03:08:43.464560 19874 tablet_replica.cc:442] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:08:43.464454 19873 tablet_replica.cc:439] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } }
I20260807 03:08:43.464735 19873 tablet_replica.cc:442] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:08:43.465214 19867 ts_tablet_manager.cc:1428] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 03:08:43.710564 19874 raft_consensus.cc:482] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:08:43.710800 19874 raft_consensus.cc:504] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:43.712323 19874 leader_election.cc:288] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19aed34633f6458f899019e9f54fcdc9 (127.17.31.1:44901), 6341165f95f14c2bb7df240e548bc06e (127.17.31.2:43991)
I20260807 03:08:43.720543 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.712918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:52427 (local address 127.17.31.2:43991)
0807 03:08:43.713213 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:43.713221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:43.713792 (+ 571us) server_negotiation.cc:407] Connection header received
0807 03:08:43.713891 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:43.713896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:43.713975 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:43.714062 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:43.714675 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.715844 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.718461 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.719653 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.719787 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.719890 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:43.720171 (+ 281us) server_negotiation.cc:299] Negotiation successful
0807 03:08:43.720267 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260807 03:08:43.720681 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.712609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:43991 (local address 127.17.31.3:52427)
0807 03:08:43.713684 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:43.713698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:43.713798 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:43.714094 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:43.714098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:43.714126 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:43.714526 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.714539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.715966 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.715975 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.718316 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.718331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.719805 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.719810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.719950 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.720017 (+ 67us) client_negotiation.cc:769] Sending connection context
0807 03:08:43.720127 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 03:08:43.720272 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":180,"spinlock_wait_cycles":2944,"thread_start_us":412,"threads_started":1}
I20260807 03:08:43.721092 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.712767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:58691 (local address 127.17.31.1:44901)
0807 03:08:43.713014 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:43.713023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:43.713486 (+ 463us) server_negotiation.cc:407] Connection header received
0807 03:08:43.713774 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:43.713780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:43.713880 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:43.713977 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:43.714522 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.715761 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.718674 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.719892 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.720020 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.720116 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:43.720458 (+ 342us) server_negotiation.cc:299] Negotiation successful
0807 03:08:43.720616 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 03:08:43.721108 19891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.712609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:44901 (local address 127.17.31.3:58691)
0807 03:08:43.713356 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:43.713373 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:43.713587 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:43.713998 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:43.714001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:43.714020 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:43.714347 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.714363 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.715916 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.715924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.718316 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.718331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.720110 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.720118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.720296 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.720326 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 03:08:43.720419 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 03:08:43.720546 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":164,"spinlock_wait_cycles":28032,"thread_start_us":784,"threads_started":1}
I20260807 03:08:43.721573 19652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc24dde9fb249ee8e88c5491c28b141" candidate_uuid: "81fb0785479c4ffe99953381648d2e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6341165f95f14c2bb7df240e548bc06e" is_pre_election: true
I20260807 03:08:43.721854 19652 raft_consensus.cc:2454] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81fb0785479c4ffe99953381648d2e97 in term 0.
I20260807 03:08:43.722280 19721 leader_election.cc:302] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6341165f95f14c2bb7df240e548bc06e, 81fb0785479c4ffe99953381648d2e97; no voters:
I20260807 03:08:43.722205 19503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc24dde9fb249ee8e88c5491c28b141" candidate_uuid: "81fb0785479c4ffe99953381648d2e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19aed34633f6458f899019e9f54fcdc9" is_pre_election: true
I20260807 03:08:43.722712 19874 raft_consensus.cc:2781] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:08:43.722710 19503 raft_consensus.cc:2454] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81fb0785479c4ffe99953381648d2e97 in term 0.
I20260807 03:08:43.722844 19874 raft_consensus.cc:482] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:08:43.722941 19874 raft_consensus.cc:3037] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:43.725091 19874 raft_consensus.cc:504] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:43.725832 19874 leader_election.cc:288] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [CANDIDATE]: Term 1 election: Requested vote from peers 19aed34633f6458f899019e9f54fcdc9 (127.17.31.1:44901), 6341165f95f14c2bb7df240e548bc06e (127.17.31.2:43991)
I20260807 03:08:43.726264 19652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc24dde9fb249ee8e88c5491c28b141" candidate_uuid: "81fb0785479c4ffe99953381648d2e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6341165f95f14c2bb7df240e548bc06e"
I20260807 03:08:43.726544 19652 raft_consensus.cc:3037] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:43.728494 19652 raft_consensus.cc:2454] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81fb0785479c4ffe99953381648d2e97 in term 1.
I20260807 03:08:43.728832 19503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bc24dde9fb249ee8e88c5491c28b141" candidate_uuid: "81fb0785479c4ffe99953381648d2e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19aed34633f6458f899019e9f54fcdc9"
I20260807 03:08:43.728863 19721 leader_election.cc:302] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6341165f95f14c2bb7df240e548bc06e, 81fb0785479c4ffe99953381648d2e97; no voters:
I20260807 03:08:43.729032 19503 raft_consensus.cc:3037] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:43.729159 19874 raft_consensus.cc:2781] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:08:43.729648 19874 raft_consensus.cc:686] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [term 1 LEADER]: Becoming Leader. State: Replica: 81fb0785479c4ffe99953381648d2e97, State: Running, Role: LEADER
I20260807 03:08:43.730190 19874 consensus_queue.cc:237] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:43.732147 19503 raft_consensus.cc:2454] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81fb0785479c4ffe99953381648d2e97 in term 1.
I20260807 03:08:43.734172 19892 tablet_replica.cc:439] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97: TxnStatusTablet state changed. Reason: New leader 81fb0785479c4ffe99953381648d2e97. Latest consensus state: current_term: 1 leader_uuid: "81fb0785479c4ffe99953381648d2e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } }
I20260807 03:08:43.734514 19892 tablet_replica.cc:442] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97: This TxnStatusTablet replica's current role is: LEADER
I20260807 03:08:43.735244 19894 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 03:08:43.735482 19894 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 03:08:43.735751 19335 catalog_manager.cc:5495] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81fb0785479c4ffe99953381648d2e97 (127.17.31.3). New cstate: current_term: 1 leader_uuid: "81fb0785479c4ffe99953381648d2e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } health_report { overall_health: HEALTHY } } }
I20260807 03:08:43.758211 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.752090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50334 (local address 127.17.31.3:39065)
0807 03:08:43.752235 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:43.752239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:43.752283 (+ 44us) server_negotiation.cc:407] Connection header received
0807 03:08:43.752569 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:43.752572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:43.752638 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:43.752747 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:08:43.753194 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.754013 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.755929 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.756431 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.756522 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.757060 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:08:43.757589 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:08:43.757695 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:43.757847 (+ 152us) server_negotiation.cc:299] Negotiation successful
0807 03:08:43.757910 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 03:08:43.758277 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.751938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:39065 (local address 127.0.0.1:50334)
0807 03:08:43.752208 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:43.752228 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:43.752381 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:43.752800 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:43.752806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:43.752819 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 03:08:43.753024 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.753036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.754144 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.754150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.755799 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.755814 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.756592 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.756599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.756815 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.756891 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 03:08:43.757701 (+ 810us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 03:08:43.757723 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 03:08:43.757813 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 03:08:43.757969 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260807 03:08:43.765120 19503 raft_consensus.cc:1261] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9 [term 1 FOLLOWER]: Refusing update from remote peer 81fb0785479c4ffe99953381648d2e97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:08:43.765148 19652 raft_consensus.cc:1261] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e [term 1 FOLLOWER]: Refusing update from remote peer 81fb0785479c4ffe99953381648d2e97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:08:43.766072 19874 consensus_queue.cc:1035] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:08:43.766449 19892 consensus_queue.cc:1035] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:08:43.776240 19873 tablet_replica.cc:439] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9: TxnStatusTablet state changed. Reason: New leader 81fb0785479c4ffe99953381648d2e97. Latest consensus state: current_term: 1 leader_uuid: "81fb0785479c4ffe99953381648d2e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } }
I20260807 03:08:43.776558 19873 tablet_replica.cc:442] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:08:43.778808 19875 tablet_replica.cc:439] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e: TxnStatusTablet state changed. Reason: New leader 81fb0785479c4ffe99953381648d2e97. Latest consensus state: current_term: 1 leader_uuid: "81fb0785479c4ffe99953381648d2e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } }
I20260807 03:08:43.779101 19875 tablet_replica.cc:442] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:08:43.782500 19897 mvcc.cc:205] Tried to move back new op lower bound from 7315751418933325824 to 7315751418803585024. Current Snapshot: MvccSnapshot[applied={T|T < 7315751418933325824}]
I20260807 03:08:43.785863 19873 tablet_replica.cc:439] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81fb0785479c4ffe99953381648d2e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } }
I20260807 03:08:43.786140 19873 tablet_replica.cc:442] T 0bc24dde9fb249ee8e88c5491c28b141 P 19aed34633f6458f899019e9f54fcdc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:08:43.790993 19892 tablet_replica.cc:439] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81fb0785479c4ffe99953381648d2e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } }
I20260807 03:08:43.791298 19892 tablet_replica.cc:442] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97: This TxnStatusTablet replica's current role is: LEADER
I20260807 03:08:43.791550 19875 tablet_replica.cc:439] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81fb0785479c4ffe99953381648d2e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } }
I20260807 03:08:43.791811 19875 tablet_replica.cc:442] T 0bc24dde9fb249ee8e88c5491c28b141 P 6341165f95f14c2bb7df240e548bc06e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:08:43.793937 19902 tablet_replica.cc:439] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81fb0785479c4ffe99953381648d2e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } } }
I20260807 03:08:43.794345 19902 tablet_replica.cc:442] T 0bc24dde9fb249ee8e88c5491c28b141 P 81fb0785479c4ffe99953381648d2e97: This TxnStatusTablet replica's current role is: LEADER
I20260807 03:08:43.806102 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.800248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.17.31.1:44901)
0807 03:08:43.800400 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:43.800405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:43.800892 (+ 487us) server_negotiation.cc:407] Connection header received
0807 03:08:43.801128 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:43.801133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:43.801229 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:43.801373 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:08:43.801844 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.802688 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.804045 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.804435 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.804580 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.804977 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:08:43.805475 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:08:43.805625 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:43.805734 (+ 109us) server_negotiation.cc:299] Negotiation successful
0807 03:08:43.805809 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 03:08:43.821950 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.816263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:58815 (local address 127.17.31.3:39065)
0807 03:08:43.816407 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:43.816412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:43.816741 (+ 329us) server_negotiation.cc:407] Connection header received
0807 03:08:43.816897 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:43.816901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:43.816978 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:43.817055 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:08:43.817615 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.818288 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.820346 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.820772 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.820866 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.821395 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:08:43.821485 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:08:43.821545 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:43.821700 (+ 155us) server_negotiation.cc:299] Negotiation successful
0807 03:08:43.821755 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260807 03:08:43.822180 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.816071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:39065 (local address 127.17.31.1:58815)
0807 03:08:43.816633 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:43.816652 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:43.816786 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:43.817098 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:43.817104 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:43.817118 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 03:08:43.817470 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.817490 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.818460 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.818466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.820217 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.820234 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.820947 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.820954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.821126 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.821196 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 03:08:43.821578 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 03:08:43.821590 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 03:08:43.821677 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 03:08:43.821815 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":323,"threads_started":1}
I20260807 03:08:43.830735 19652 raft_consensus.cc:1261] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e [term 1 FOLLOWER]: Refusing update from remote peer 19aed34633f6458f899019e9f54fcdc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:08:43.830863 19799 raft_consensus.cc:1261] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Refusing update from remote peer 19aed34633f6458f899019e9f54fcdc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:08:43.831537 19878 consensus_queue.cc:1035] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:08:43.831887 19873 consensus_queue.cc:1035] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:08:43.837809 19898 mvcc.cc:205] Tried to move back new op lower bound from 7315751419204640768 to 7315751417342599168. Current Snapshot: MvccSnapshot[applied={T|T < 7315751419204640768}]
I20260807 03:08:43.843509 19899 mvcc.cc:205] Tried to move back new op lower bound from 7315751419204640768 to 7315751417342599168. Current Snapshot: MvccSnapshot[applied={T|T < 7315751419204640768}]
I20260807 03:08:43.867168 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.860331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:45073 (local address 127.17.31.1:44901)
0807 03:08:43.860471 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:43.860476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:43.861142 (+ 666us) server_negotiation.cc:407] Connection header received
0807 03:08:43.861348 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:43.861354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:43.861441 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:43.861538 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:08:43.862049 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.863128 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.864796 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.865387 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.865531 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.866105 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:08:43.866239 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:08:43.866361 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:43.866796 (+ 435us) server_negotiation.cc:299] Negotiation successful
0807 03:08:43.866878 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 03:08:43.867914 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:43.860407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:44901 (local address 127.17.31.3:45073)
0807 03:08:43.861040 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:43.861059 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:43.861215 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:43.861563 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:43.861567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:43.861581 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 03:08:43.861898 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.861907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.863271 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.863283 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.864607 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:43.864618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:43.865587 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:43.865594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:43.865828 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:43.865921 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 03:08:43.866365 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 03:08:43.866377 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 03:08:43.867348 (+ 971us) client_negotiation.cc:240] Negotiation successful
0807 03:08:43.867512 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":340,"threads_started":1}
I20260807 03:08:43.905422 19335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:08:43.905727 19335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:08:43.909936 19335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97: Sending DeleteTablet for 3 replicas of tablet be1cb2f50f514bdda6cb8dc1179446ff
I20260807 03:08:43.911360 19628 tablet_service.cc:1514] Processing DeleteTablet for tablet be1cb2f50f514bdda6cb8dc1179446ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:43 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:34206
I20260807 03:08:43.911887 19772 tablet_service.cc:1514] Processing DeleteTablet for tablet be1cb2f50f514bdda6cb8dc1179446ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:43 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:50320
I20260807 03:08:43.912267 19481 tablet_service.cc:1514] Processing DeleteTablet for tablet be1cb2f50f514bdda6cb8dc1179446ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:43 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:37482
I20260807 03:08:43.919685 19335 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 03:08:43.920208 19918 tablet_replica.cc:330] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97: stopping tablet replica
I20260807 03:08:43.920699 19918 raft_consensus.cc:2229] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:08:43.921036 19918 raft_consensus.cc:2258] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260807 03:08:43.921674 19335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:08:43.923022 19918 ts_tablet_manager.cc:1905] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:43.931121 19918 ts_tablet_manager.cc:1918] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 03:08:43.931290 19918 log.cc:1196] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/be1cb2f50f514bdda6cb8dc1179446ff
I20260807 03:08:43.931824 19918 ts_tablet_manager.cc:1939] T be1cb2f50f514bdda6cb8dc1179446ff P 81fb0785479c4ffe99953381648d2e97: Deleting consensus metadata
I20260807 03:08:43.933573 19321 catalog_manager.cc:4841] TS 81fb0785479c4ffe99953381648d2e97 (127.17.31.3:39065): tablet be1cb2f50f514bdda6cb8dc1179446ff (table test-table [id=8a387503cde44e97b0a8a028dd28e924]) successfully deleted
I20260807 03:08:43.934486 19772 tablet_service.cc:1467] Processing CreateTablet for tablet f546cd4597d747cdb2afed6b0594d7bb (DEFAULT_TABLE table=test-table [id=6d56270256fa4a8b8c10594feb4d64c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:43.934914 19772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f546cd4597d747cdb2afed6b0594d7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:43.935082 19480 tablet_service.cc:1467] Processing CreateTablet for tablet f546cd4597d747cdb2afed6b0594d7bb (DEFAULT_TABLE table=test-table [id=6d56270256fa4a8b8c10594feb4d64c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:43.935631 19480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f546cd4597d747cdb2afed6b0594d7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:43.938668 19627 tablet_service.cc:1467] Processing CreateTablet for tablet f546cd4597d747cdb2afed6b0594d7bb (DEFAULT_TABLE table=test-table [id=6d56270256fa4a8b8c10594feb4d64c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:43.939229 19627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f546cd4597d747cdb2afed6b0594d7bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:43.942826 19867 tablet_bootstrap.cc:492] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9: Bootstrap starting.
I20260807 03:08:43.944604 19867 tablet_bootstrap.cc:654] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:43.947714 19919 tablet_replica.cc:330] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9: stopping tablet replica
I20260807 03:08:43.948307 19919 raft_consensus.cc:2229] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:08:43.948925 19917 tablet_replica.cc:330] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e: stopping tablet replica
I20260807 03:08:43.949586 19919 raft_consensus.cc:2258] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:43.950099 19917 raft_consensus.cc:2229] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:08:43.950738 19917 raft_consensus.cc:2258] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:43.952472 19919 ts_tablet_manager.cc:1905] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:43.954267 19917 ts_tablet_manager.cc:1905] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:43.958173 19865 tablet_bootstrap.cc:492] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97: Bootstrap starting.
I20260807 03:08:43.959877 19865 tablet_bootstrap.cc:654] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:43.966674 19917 ts_tablet_manager.cc:1918] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 03:08:43.970804 19919 ts_tablet_manager.cc:1918] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 03:08:43.970716 19866 tablet_bootstrap.cc:492] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e: Bootstrap starting.
I20260807 03:08:43.972779 19866 tablet_bootstrap.cc:654] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:43.977450 19917 log.cc:1196] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/be1cb2f50f514bdda6cb8dc1179446ff
I20260807 03:08:43.978199 19917 ts_tablet_manager.cc:1939] T be1cb2f50f514bdda6cb8dc1179446ff P 6341165f95f14c2bb7df240e548bc06e: Deleting consensus metadata
I20260807 03:08:43.978655 19919 log.cc:1196] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/be1cb2f50f514bdda6cb8dc1179446ff
I20260807 03:08:43.979456 19919 ts_tablet_manager.cc:1939] T be1cb2f50f514bdda6cb8dc1179446ff P 19aed34633f6458f899019e9f54fcdc9: Deleting consensus metadata
I20260807 03:08:43.981662 19322 catalog_manager.cc:4841] TS 6341165f95f14c2bb7df240e548bc06e (127.17.31.2:43991): tablet be1cb2f50f514bdda6cb8dc1179446ff (table test-table [id=8a387503cde44e97b0a8a028dd28e924]) successfully deleted
I20260807 03:08:43.982110 19867 tablet_bootstrap.cc:492] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9: No bootstrap required, opened a new log
I20260807 03:08:43.982415 19867 ts_tablet_manager.cc:1397] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.005s
I20260807 03:08:43.983264 19867 raft_consensus.cc:348] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:43.983705 19324 catalog_manager.cc:4841] TS 19aed34633f6458f899019e9f54fcdc9 (127.17.31.1:44901): tablet be1cb2f50f514bdda6cb8dc1179446ff (table test-table [id=8a387503cde44e97b0a8a028dd28e924]) successfully deleted
I20260807 03:08:43.984596 19865 tablet_bootstrap.cc:492] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97: No bootstrap required, opened a new log
I20260807 03:08:43.984841 19865 ts_tablet_manager.cc:1397] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.008s
I20260807 03:08:43.985716 19865 raft_consensus.cc:348] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:43.986020 19865 raft_consensus.cc:374] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:43.986140 19865 raft_consensus.cc:729] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81fb0785479c4ffe99953381648d2e97, State: Initialized, Role: FOLLOWER
I20260807 03:08:43.986495 19865 consensus_queue.cc:260] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:43.987823 19866 tablet_bootstrap.cc:492] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e: No bootstrap required, opened a new log
I20260807 03:08:43.988029 19866 ts_tablet_manager.cc:1397] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20260807 03:08:43.988873 19866 raft_consensus.cc:348] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:43.989166 19866 raft_consensus.cc:374] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:43.989295 19866 raft_consensus.cc:729] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6341165f95f14c2bb7df240e548bc06e, State: Initialized, Role: FOLLOWER
I20260807 03:08:43.989612 19866 consensus_queue.cc:260] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:43.990684 19865 ts_tablet_manager.cc:1428] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260807 03:08:43.991138 19866 ts_tablet_manager.cc:1428] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 03:08:43.993745 19867 raft_consensus.cc:374] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:43.993911 19867 raft_consensus.cc:729] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19aed34633f6458f899019e9f54fcdc9, State: Initialized, Role: FOLLOWER
I20260807 03:08:43.994197 19867 consensus_queue.cc:260] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:43.995019 19867 ts_tablet_manager.cc:1428] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20260807 03:08:44.024964 19875 raft_consensus.cc:482] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:08:44.025205 19875 raft_consensus.cc:504] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:44.027071 19875 leader_election.cc:288] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19aed34633f6458f899019e9f54fcdc9 (127.17.31.1:44901), 81fb0785479c4ffe99953381648d2e97 (127.17.31.3:39065)
I20260807 03:08:44.048275 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:44.027978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:46809 (local address 127.17.31.3:39065)
0807 03:08:44.028388 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:44.028398 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:44.040113 (+ 11715us) server_negotiation.cc:407] Connection header received
0807 03:08:44.040565 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:44.040570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:44.040652 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:44.040713 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:44.041620 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.042835 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.045552 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.046982 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.047108 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:44.047211 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:44.047804 (+ 593us) server_negotiation.cc:299] Negotiation successful
0807 03:08:44.047940 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260807 03:08:44.048606 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:44.027848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:49439 (local address 127.17.31.1:44901)
0807 03:08:44.028220 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:44.028227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:44.039041 (+ 10814us) server_negotiation.cc:407] Connection header received
0807 03:08:44.039508 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:44.039514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:44.039594 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:44.039654 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:08:44.040498 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.041682 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.044693 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.046074 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.046169 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:44.046269 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:44.048124 (+ 1855us) server_negotiation.cc:299] Negotiation successful
0807 03:08:44.048265 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260807 03:08:44.049471 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:44.027452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:44901 (local address 127.17.31.2:49439)
0807 03:08:44.038935 (+ 11483us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:44.038956 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:44.039332 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:44.039872 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:44.039877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:44.039907 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:44.040271 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:44.040285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.041867 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.041873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:44.044506 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:44.044520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.046479 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.046486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:44.046669 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:44.046715 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 03:08:44.048927 (+ 2212us) client_negotiation.cc:240] Negotiation successful
0807 03:08:44.049065 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11311,"mutex_wait_us":1,"thread_start_us":11601,"threads_started":1}
I20260807 03:08:44.050069 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:44.027606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:39065 (local address 127.17.31.2:46809)
0807 03:08:44.039997 (+ 12391us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:44.040018 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:44.040375 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:44.040914 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:44.040919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:44.040945 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:08:44.041414 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:44.041428 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.043111 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.043116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:44.045320 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:44.045337 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.047422 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.047428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:44.047607 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:44.047640 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 03:08:44.048773 (+ 1133us) client_negotiation.cc:240] Negotiation successful
0807 03:08:44.049696 (+ 923us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12251,"mutex_wait_us":36,"thread_start_us":12512,"threads_started":1}
I20260807 03:08:44.050136 19503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f546cd4597d747cdb2afed6b0594d7bb" candidate_uuid: "6341165f95f14c2bb7df240e548bc06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19aed34633f6458f899019e9f54fcdc9" is_pre_election: true
I20260807 03:08:44.050410 19503 raft_consensus.cc:2454] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6341165f95f14c2bb7df240e548bc06e in term 0.
I20260807 03:08:44.050836 19799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f546cd4597d747cdb2afed6b0594d7bb" candidate_uuid: "6341165f95f14c2bb7df240e548bc06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81fb0785479c4ffe99953381648d2e97" is_pre_election: true
I20260807 03:08:44.051021 19581 leader_election.cc:302] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19aed34633f6458f899019e9f54fcdc9, 6341165f95f14c2bb7df240e548bc06e; no voters:
I20260807 03:08:44.051086 19799 raft_consensus.cc:2454] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6341165f95f14c2bb7df240e548bc06e in term 0.
I20260807 03:08:44.051398 19875 raft_consensus.cc:2781] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:08:44.051506 19875 raft_consensus.cc:482] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:08:44.051600 19875 raft_consensus.cc:3037] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:44.054651 19875 raft_consensus.cc:504] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:44.055305 19875 leader_election.cc:288] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [CANDIDATE]: Term 1 election: Requested vote from peers 19aed34633f6458f899019e9f54fcdc9 (127.17.31.1:44901), 81fb0785479c4ffe99953381648d2e97 (127.17.31.3:39065)
I20260807 03:08:44.055725 19503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f546cd4597d747cdb2afed6b0594d7bb" candidate_uuid: "6341165f95f14c2bb7df240e548bc06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19aed34633f6458f899019e9f54fcdc9"
I20260807 03:08:44.055928 19503 raft_consensus.cc:3037] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:44.055905 19799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f546cd4597d747cdb2afed6b0594d7bb" candidate_uuid: "6341165f95f14c2bb7df240e548bc06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81fb0785479c4ffe99953381648d2e97"
I20260807 03:08:44.056106 19799 raft_consensus.cc:3037] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:44.059116 19503 raft_consensus.cc:2454] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6341165f95f14c2bb7df240e548bc06e in term 1.
I20260807 03:08:44.059214 19799 raft_consensus.cc:2454] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6341165f95f14c2bb7df240e548bc06e in term 1.
I20260807 03:08:44.059574 19581 leader_election.cc:302] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19aed34633f6458f899019e9f54fcdc9, 6341165f95f14c2bb7df240e548bc06e; no voters:
I20260807 03:08:44.060140 19875 raft_consensus.cc:2781] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:08:44.060616 19875 raft_consensus.cc:686] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [term 1 LEADER]: Becoming Leader. State: Replica: 6341165f95f14c2bb7df240e548bc06e, State: Running, Role: LEADER
I20260807 03:08:44.061089 19875 consensus_queue.cc:237] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:44.068684 19335 catalog_manager.cc:5495] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6341165f95f14c2bb7df240e548bc06e (127.17.31.2). New cstate: current_term: 1 leader_uuid: "6341165f95f14c2bb7df240e548bc06e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } health_report { overall_health: UNKNOWN } } }
I20260807 03:08:44.130057 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:44.122656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.17.31.2:43991)
0807 03:08:44.122812 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:44.122817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:44.122834 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:08:44.122926 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:44.122934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:44.123029 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:44.123170 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:08:44.123744 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.124870 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.126868 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.127531 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.127655 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:44.128239 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:08:44.128916 (+ 677us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:08:44.129056 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:44.129702 (+ 646us) server_negotiation.cc:299] Negotiation successful
0807 03:08:44.129786 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 03:08:44.181797 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:44.160262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:46895 (local address 127.17.31.3:39065)
0807 03:08:44.160462 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:44.160468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:44.171539 (+ 11071us) server_negotiation.cc:407] Connection header received
0807 03:08:44.172191 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:44.172198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:44.172319 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:44.172422 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:08:44.173789 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.175394 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.178471 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.179558 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.179664 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:44.180695 (+ 1031us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:08:44.180870 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:08:44.180949 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:44.181244 (+ 295us) server_negotiation.cc:299] Negotiation successful
0807 03:08:44.181340 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 03:08:44.183475 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:44.159905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:39065 (local address 127.17.31.2:46895)
0807 03:08:44.171412 (+ 11507us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:44.171435 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:44.171912 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:44.172691 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:44.172696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:44.172720 (+ 24us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 03:08:44.173502 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:44.173520 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.175622 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.175628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:44.178237 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:44.178252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.180024 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.180032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:44.180357 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:44.180430 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 03:08:44.181096 (+ 666us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 03:08:44.181107 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 03:08:44.182427 (+ 1320us) client_negotiation.cc:240] Negotiation successful
0807 03:08:44.182637 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11173,"thread_start_us":11682,"threads_started":1}
I20260807 03:08:44.196303 19799 raft_consensus.cc:1261] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Refusing update from remote peer 6341165f95f14c2bb7df240e548bc06e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:08:44.196609 19502 raft_consensus.cc:1261] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9 [term 1 FOLLOWER]: Refusing update from remote peer 6341165f95f14c2bb7df240e548bc06e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:08:44.197160 19925 consensus_queue.cc:1035] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:08:44.197979 19925 consensus_queue.cc:1035] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:08:44.252748 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:44.242801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:43991 (local address 127.17.31.3:58199)
0807 03:08:44.243192 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:44.243214 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:44.243351 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:44.244544 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:44.244557 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:44.244579 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 03:08:44.244919 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:44.244933 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.247008 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.247013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:44.249440 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:44.249455 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.251095 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.251102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:44.251414 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:44.251464 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 03:08:44.252011 (+ 547us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 03:08:44.252023 (+ 12us) client_negotiation.cc:769] Sending connection context
0807 03:08:44.252100 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 03:08:44.252243 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244}
I20260807 03:08:44.253587 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:44.243656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:58199 (local address 127.17.31.2:43991)
0807 03:08:44.243945 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:44.243951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:44.243978 (+ 27us) server_negotiation.cc:407] Connection header received
0807 03:08:44.244055 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:44.244061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:44.244158 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:44.244283 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:08:44.245124 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.246781 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.249631 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:44.250809 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:44.250940 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:44.251695 (+ 755us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:08:44.251864 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:08:44.251956 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:44.253089 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0807 03:08:44.253168 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260807 03:08:44.255409 19899 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 03:08:45.868654 19337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:08:45.868870 19337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:08:45.872937 19337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97: Sending DeleteTablet for 3 replicas of tablet f546cd4597d747cdb2afed6b0594d7bb
I20260807 03:08:45.873621 19481 tablet_service.cc:1514] Processing DeleteTablet for tablet f546cd4597d747cdb2afed6b0594d7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:45 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:37482
I20260807 03:08:45.873865 19627 tablet_service.cc:1514] Processing DeleteTablet for tablet f546cd4597d747cdb2afed6b0594d7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:45 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:34206
I20260807 03:08:45.874040 19772 tablet_service.cc:1514] Processing DeleteTablet for tablet f546cd4597d747cdb2afed6b0594d7bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:45 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:50320
I20260807 03:08:45.886652 19956 tablet_replica.cc:330] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9: stopping tablet replica
I20260807 03:08:45.886837 19957 tablet_replica.cc:330] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e: stopping tablet replica
I20260807 03:08:45.886940 19956 raft_consensus.cc:2229] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:08:45.887120 19957 raft_consensus.cc:2229] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:08:45.887404 19956 raft_consensus.cc:2258] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:45.887769 19957 raft_consensus.cc:2258] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:45.888602 19956 ts_tablet_manager.cc:1905] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:45.888671 19957 ts_tablet_manager.cc:1905] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:45.889573 19958 tablet_replica.cc:330] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97: stopping tablet replica
I20260807 03:08:45.889871 19958 raft_consensus.cc:2229] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:08:45.890177 19958 raft_consensus.cc:2258] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:45.891330 19958 ts_tablet_manager.cc:1905] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:45.896597 19957 ts_tablet_manager.cc:1918] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 03:08:45.896597 19956 ts_tablet_manager.cc:1918] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 03:08:45.896802 19956 log.cc:1196] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/f546cd4597d747cdb2afed6b0594d7bb
I20260807 03:08:45.896883 19957 log.cc:1196] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/f546cd4597d747cdb2afed6b0594d7bb
I20260807 03:08:45.897583 19956 ts_tablet_manager.cc:1939] T f546cd4597d747cdb2afed6b0594d7bb P 19aed34633f6458f899019e9f54fcdc9: Deleting consensus metadata
I20260807 03:08:45.897583 19957 ts_tablet_manager.cc:1939] T f546cd4597d747cdb2afed6b0594d7bb P 6341165f95f14c2bb7df240e548bc06e: Deleting consensus metadata
I20260807 03:08:45.897874 19958 ts_tablet_manager.cc:1918] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 03:08:45.898099 19958 log.cc:1196] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/f546cd4597d747cdb2afed6b0594d7bb
I20260807 03:08:45.898788 19324 catalog_manager.cc:4841] TS 19aed34633f6458f899019e9f54fcdc9 (127.17.31.1:44901): tablet f546cd4597d747cdb2afed6b0594d7bb (table test-table [id=6d56270256fa4a8b8c10594feb4d64c8]) successfully deleted
I20260807 03:08:45.898855 19958 ts_tablet_manager.cc:1939] T f546cd4597d747cdb2afed6b0594d7bb P 81fb0785479c4ffe99953381648d2e97: Deleting consensus metadata
I20260807 03:08:45.898874 19322 catalog_manager.cc:4841] TS 6341165f95f14c2bb7df240e548bc06e (127.17.31.2:43991): tablet f546cd4597d747cdb2afed6b0594d7bb (table test-table [id=6d56270256fa4a8b8c10594feb4d64c8]) successfully deleted
I20260807 03:08:45.899962 19321 catalog_manager.cc:4841] TS 81fb0785479c4ffe99953381648d2e97 (127.17.31.3:39065): tablet f546cd4597d747cdb2afed6b0594d7bb (table test-table [id=6d56270256fa4a8b8c10594feb4d64c8]) successfully deleted
Aug 07 03:08:45 dist-test-slave-tm1g krb5kdc[19292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072125, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 03:08:45 dist-test-slave-tm1g krb5kdc[19292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072125, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:45.944782 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:45.927430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.17.31.62:46185)
0807 03:08:45.928078 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:45.928082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:45.928104 (+ 22us) server_negotiation.cc:407] Connection header received
0807 03:08:45.928233 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:45.928236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:45.928300 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:45.928390 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:45.930169 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:45.931125 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:45.932599 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:45.933200 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:45.933294 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:45.938494 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:45.938500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:45.938504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:45.938534 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:45.942068 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:45.942821 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:45.942825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:45.942827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:45.942987 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:45.943336 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:45.943339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:45.943341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:45.943675 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:45.943863 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:45.944107 (+ 244us) server_negotiation.cc:299] Negotiation successful
0807 03:08:45.944265 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":350,"threads_started":1}
I20260807 03:08:45.948402 19337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:08:45.949182 19337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:08:45.955730 19627 tablet_service.cc:1467] Processing CreateTablet for tablet ea2cc74fa9f641b5b1f71e4b31a7106b (DEFAULT_TABLE table=test-table [id=f4bb1737a0c8483a80c4703420e8be50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:45.955695 19481 tablet_service.cc:1467] Processing CreateTablet for tablet ea2cc74fa9f641b5b1f71e4b31a7106b (DEFAULT_TABLE table=test-table [id=f4bb1737a0c8483a80c4703420e8be50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:45.955832 19772 tablet_service.cc:1467] Processing CreateTablet for tablet ea2cc74fa9f641b5b1f71e4b31a7106b (DEFAULT_TABLE table=test-table [id=f4bb1737a0c8483a80c4703420e8be50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:45.956336 19481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2cc74fa9f641b5b1f71e4b31a7106b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:45.956463 19772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2cc74fa9f641b5b1f71e4b31a7106b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:45.956461 19627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2cc74fa9f641b5b1f71e4b31a7106b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:45.963495 19968 tablet_bootstrap.cc:492] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9: Bootstrap starting.
I20260807 03:08:45.965534 19969 tablet_bootstrap.cc:492] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e: Bootstrap starting.
I20260807 03:08:45.966895 19968 tablet_bootstrap.cc:654] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:45.967382 19970 tablet_bootstrap.cc:492] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97: Bootstrap starting.
I20260807 03:08:45.968658 19969 tablet_bootstrap.cc:654] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:45.969918 19970 tablet_bootstrap.cc:654] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:45.970918 19968 tablet_bootstrap.cc:492] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9: No bootstrap required, opened a new log
I20260807 03:08:45.971146 19968 ts_tablet_manager.cc:1397] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260807 03:08:45.972240 19969 tablet_bootstrap.cc:492] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e: No bootstrap required, opened a new log
I20260807 03:08:45.972141 19968 raft_consensus.cc:348] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:45.972484 19968 raft_consensus.cc:374] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:45.972688 19968 raft_consensus.cc:729] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19aed34633f6458f899019e9f54fcdc9, State: Initialized, Role: FOLLOWER
I20260807 03:08:45.973054 19969 ts_tablet_manager.cc:1397] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260807 03:08:45.973004 19968 consensus_queue.cc:260] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:45.973722 19970 tablet_bootstrap.cc:492] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97: No bootstrap required, opened a new log
I20260807 03:08:45.974053 19970 ts_tablet_manager.cc:1397] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 03:08:45.974241 19969 raft_consensus.cc:348] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:45.974539 19968 ts_tablet_manager.cc:1428] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260807 03:08:45.974659 19969 raft_consensus.cc:374] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:45.974907 19969 raft_consensus.cc:729] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6341165f95f14c2bb7df240e548bc06e, State: Initialized, Role: FOLLOWER
I20260807 03:08:45.975265 19969 consensus_queue.cc:260] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:45.975666 19970 raft_consensus.cc:348] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:45.976070 19970 raft_consensus.cc:374] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:45.976084 19969 ts_tablet_manager.cc:1428] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 03:08:45.976325 19970 raft_consensus.cc:729] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81fb0785479c4ffe99953381648d2e97, State: Initialized, Role: FOLLOWER
I20260807 03:08:45.976643 19970 consensus_queue.cc:260] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:45.977458 19970 ts_tablet_manager.cc:1428] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260807 03:08:46.135581 19974 raft_consensus.cc:482] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:08:46.136023 19974 raft_consensus.cc:504] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:46.136981 19974 leader_election.cc:288] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6341165f95f14c2bb7df240e548bc06e (127.17.31.2:43991), 81fb0785479c4ffe99953381648d2e97 (127.17.31.3:39065)
I20260807 03:08:46.137662 19798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2cc74fa9f641b5b1f71e4b31a7106b" candidate_uuid: "19aed34633f6458f899019e9f54fcdc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81fb0785479c4ffe99953381648d2e97" is_pre_election: true
I20260807 03:08:46.137663 19652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2cc74fa9f641b5b1f71e4b31a7106b" candidate_uuid: "19aed34633f6458f899019e9f54fcdc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6341165f95f14c2bb7df240e548bc06e" is_pre_election: true
I20260807 03:08:46.137962 19798 raft_consensus.cc:2454] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19aed34633f6458f899019e9f54fcdc9 in term 0.
I20260807 03:08:46.137962 19652 raft_consensus.cc:2454] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19aed34633f6458f899019e9f54fcdc9 in term 0.
I20260807 03:08:46.138547 19429 leader_election.cc:302] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19aed34633f6458f899019e9f54fcdc9, 81fb0785479c4ffe99953381648d2e97; no voters:
I20260807 03:08:46.138875 19974 raft_consensus.cc:2781] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:08:46.139055 19974 raft_consensus.cc:482] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:08:46.139228 19974 raft_consensus.cc:3037] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:46.141819 19974 raft_consensus.cc:504] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:46.142484 19974 leader_election.cc:288] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [CANDIDATE]: Term 1 election: Requested vote from peers 6341165f95f14c2bb7df240e548bc06e (127.17.31.2:43991), 81fb0785479c4ffe99953381648d2e97 (127.17.31.3:39065)
I20260807 03:08:46.143059 19652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2cc74fa9f641b5b1f71e4b31a7106b" candidate_uuid: "19aed34633f6458f899019e9f54fcdc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6341165f95f14c2bb7df240e548bc06e"
I20260807 03:08:46.143070 19798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2cc74fa9f641b5b1f71e4b31a7106b" candidate_uuid: "19aed34633f6458f899019e9f54fcdc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81fb0785479c4ffe99953381648d2e97"
I20260807 03:08:46.143281 19652 raft_consensus.cc:3037] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:46.143308 19798 raft_consensus.cc:3037] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:46.145831 19798 raft_consensus.cc:2454] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19aed34633f6458f899019e9f54fcdc9 in term 1.
I20260807 03:08:46.145833 19652 raft_consensus.cc:2454] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19aed34633f6458f899019e9f54fcdc9 in term 1.
I20260807 03:08:46.146248 19430 leader_election.cc:302] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19aed34633f6458f899019e9f54fcdc9, 6341165f95f14c2bb7df240e548bc06e; no voters:
I20260807 03:08:46.146745 19974 raft_consensus.cc:2781] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:08:46.147682 19974 raft_consensus.cc:686] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [term 1 LEADER]: Becoming Leader. State: Replica: 19aed34633f6458f899019e9f54fcdc9, State: Running, Role: LEADER
I20260807 03:08:46.148080 19974 consensus_queue.cc:237] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:46.151204 19337 catalog_manager.cc:5495] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19aed34633f6458f899019e9f54fcdc9 (127.17.31.1). New cstate: current_term: 1 leader_uuid: "19aed34633f6458f899019e9f54fcdc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } health_report { overall_health: UNKNOWN } } }
I20260807 03:08:46.213706 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:46.206410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.17.31.1:44901)
0807 03:08:46.207726 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:46.207730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:46.207758 (+ 28us) server_negotiation.cc:407] Connection header received
0807 03:08:46.207805 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:46.207808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:46.207865 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:46.207986 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:08:46.208466 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:46.209668 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:46.211291 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:46.211997 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:46.212132 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:46.212750 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:08:46.212867 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:08:46.212988 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:46.213130 (+ 142us) server_negotiation.cc:299] Negotiation successful
0807 03:08:46.213188 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1118,"thread_start_us":947,"threads_started":1}
I20260807 03:08:46.223830 19798 raft_consensus.cc:1261] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Refusing update from remote peer 19aed34633f6458f899019e9f54fcdc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:08:46.223866 19652 raft_consensus.cc:1261] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e [term 1 FOLLOWER]: Refusing update from remote peer 19aed34633f6458f899019e9f54fcdc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:08:46.224506 19975 consensus_queue.cc:1035] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:08:46.224987 19974 consensus_queue.cc:1035] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:08:46.274782 19335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:08:46.275094 19335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:08:46.278028 19335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97: Sending DeleteTablet for 3 replicas of tablet ea2cc74fa9f641b5b1f71e4b31a7106b
I20260807 03:08:46.279022 19481 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2cc74fa9f641b5b1f71e4b31a7106b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:46 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:37482
I20260807 03:08:46.279292 19772 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2cc74fa9f641b5b1f71e4b31a7106b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:46 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:50320
I20260807 03:08:46.279429 19627 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2cc74fa9f641b5b1f71e4b31a7106b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:46 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:34206
I20260807 03:08:46.279678 19958 tablet_replica.cc:330] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97: stopping tablet replica
I20260807 03:08:46.279750 19957 tablet_replica.cc:330] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e: stopping tablet replica
I20260807 03:08:46.279768 19956 tablet_replica.cc:330] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9: stopping tablet replica
I20260807 03:08:46.279971 19958 raft_consensus.cc:2229] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:08:46.280056 19957 raft_consensus.cc:2229] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:08:46.280056 19956 raft_consensus.cc:2229] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:08:46.280217 19958 raft_consensus.cc:2258] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:46.280283 19957 raft_consensus.cc:2258] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:46.280436 19956 raft_consensus.cc:2258] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:46.281421 19958 ts_tablet_manager.cc:1905] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:46.281488 19956 ts_tablet_manager.cc:1905] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:46.281625 19957 ts_tablet_manager.cc:1905] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:46.286100 19958 ts_tablet_manager.cc:1918] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 03:08:46.286249 19958 log.cc:1196] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/ea2cc74fa9f641b5b1f71e4b31a7106b
I20260807 03:08:46.286654 19335 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 03:08:46.287010 19958 ts_tablet_manager.cc:1939] T ea2cc74fa9f641b5b1f71e4b31a7106b P 81fb0785479c4ffe99953381648d2e97: Deleting consensus metadata
W20260807 03:08:46.287472 19335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:08:46.288600 19957 ts_tablet_manager.cc:1918] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 03:08:46.288841 19957 log.cc:1196] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/ea2cc74fa9f641b5b1f71e4b31a7106b
I20260807 03:08:46.288848 19956 ts_tablet_manager.cc:1918] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 03:08:46.288987 19956 log.cc:1196] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/ea2cc74fa9f641b5b1f71e4b31a7106b
I20260807 03:08:46.289449 19957 ts_tablet_manager.cc:1939] T ea2cc74fa9f641b5b1f71e4b31a7106b P 6341165f95f14c2bb7df240e548bc06e: Deleting consensus metadata
I20260807 03:08:46.289608 19956 ts_tablet_manager.cc:1939] T ea2cc74fa9f641b5b1f71e4b31a7106b P 19aed34633f6458f899019e9f54fcdc9: Deleting consensus metadata
I20260807 03:08:46.290516 19321 catalog_manager.cc:4841] TS 81fb0785479c4ffe99953381648d2e97 (127.17.31.3:39065): tablet ea2cc74fa9f641b5b1f71e4b31a7106b (table test-table [id=f4bb1737a0c8483a80c4703420e8be50]) successfully deleted
I20260807 03:08:46.290624 19322 catalog_manager.cc:4841] TS 6341165f95f14c2bb7df240e548bc06e (127.17.31.2:43991): tablet ea2cc74fa9f641b5b1f71e4b31a7106b (table test-table [id=f4bb1737a0c8483a80c4703420e8be50]) successfully deleted
I20260807 03:08:46.290719 19324 catalog_manager.cc:4841] TS 19aed34633f6458f899019e9f54fcdc9 (127.17.31.1:44901): tablet ea2cc74fa9f641b5b1f71e4b31a7106b (table test-table [id=f4bb1737a0c8483a80c4703420e8be50]) successfully deleted
I20260807 03:08:46.294641 19481 tablet_service.cc:1467] Processing CreateTablet for tablet a50d9c2e203f40a5b2988674e573c09e (DEFAULT_TABLE table=test-table [id=65a07191dbec46c1a2de07b95a203773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:46.294919 19627 tablet_service.cc:1467] Processing CreateTablet for tablet a50d9c2e203f40a5b2988674e573c09e (DEFAULT_TABLE table=test-table [id=65a07191dbec46c1a2de07b95a203773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:46.294934 19772 tablet_service.cc:1467] Processing CreateTablet for tablet a50d9c2e203f40a5b2988674e573c09e (DEFAULT_TABLE table=test-table [id=65a07191dbec46c1a2de07b95a203773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:08:46.295110 19481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a50d9c2e203f40a5b2988674e573c09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:46.295334 19772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a50d9c2e203f40a5b2988674e573c09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:46.295349 19627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a50d9c2e203f40a5b2988674e573c09e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:46.300735 19970 tablet_bootstrap.cc:492] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97: Bootstrap starting.
I20260807 03:08:46.302986 19970 tablet_bootstrap.cc:654] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:46.303591 19969 tablet_bootstrap.cc:492] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e: Bootstrap starting.
I20260807 03:08:46.303645 19968 tablet_bootstrap.cc:492] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9: Bootstrap starting.
I20260807 03:08:46.305900 19969 tablet_bootstrap.cc:654] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:46.306488 19970 tablet_bootstrap.cc:492] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97: No bootstrap required, opened a new log
I20260807 03:08:46.306535 19968 tablet_bootstrap.cc:654] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:46.306772 19970 ts_tablet_manager.cc:1397] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260807 03:08:46.308015 19970 raft_consensus.cc:348] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:46.308612 19970 raft_consensus.cc:374] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:46.308851 19970 raft_consensus.cc:729] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81fb0785479c4ffe99953381648d2e97, State: Initialized, Role: FOLLOWER
I20260807 03:08:46.309332 19970 consensus_queue.cc:260] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:46.309962 19969 tablet_bootstrap.cc:492] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e: No bootstrap required, opened a new log
I20260807 03:08:46.310256 19969 ts_tablet_manager.cc:1397] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260807 03:08:46.310428 19970 ts_tablet_manager.cc:1428] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260807 03:08:46.311205 19969 raft_consensus.cc:348] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:46.311745 19969 raft_consensus.cc:374] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:46.311877 19969 raft_consensus.cc:729] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6341165f95f14c2bb7df240e548bc06e, State: Initialized, Role: FOLLOWER
I20260807 03:08:46.312252 19969 consensus_queue.cc:260] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:46.313406 19968 tablet_bootstrap.cc:492] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9: No bootstrap required, opened a new log
I20260807 03:08:46.313469 19969 ts_tablet_manager.cc:1428] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260807 03:08:46.313688 19968 ts_tablet_manager.cc:1397] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260807 03:08:46.314772 19968 raft_consensus.cc:348] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:46.315095 19968 raft_consensus.cc:374] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:46.315230 19968 raft_consensus.cc:729] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19aed34633f6458f899019e9f54fcdc9, State: Initialized, Role: FOLLOWER
I20260807 03:08:46.315550 19968 consensus_queue.cc:260] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:46.316175 19968 ts_tablet_manager.cc:1428] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260807 03:08:46.369788 19948 raft_consensus.cc:482] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:08:46.370004 19948 raft_consensus.cc:504] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:46.370822 19948 leader_election.cc:288] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19aed34633f6458f899019e9f54fcdc9 (127.17.31.1:44901), 81fb0785479c4ffe99953381648d2e97 (127.17.31.3:39065)
I20260807 03:08:46.371328 19503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50d9c2e203f40a5b2988674e573c09e" candidate_uuid: "6341165f95f14c2bb7df240e548bc06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19aed34633f6458f899019e9f54fcdc9" is_pre_election: true
I20260807 03:08:46.371374 19798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50d9c2e203f40a5b2988674e573c09e" candidate_uuid: "6341165f95f14c2bb7df240e548bc06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81fb0785479c4ffe99953381648d2e97" is_pre_election: true
I20260807 03:08:46.371711 19503 raft_consensus.cc:2454] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6341165f95f14c2bb7df240e548bc06e in term 0.
I20260807 03:08:46.371739 19798 raft_consensus.cc:2454] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6341165f95f14c2bb7df240e548bc06e in term 0.
I20260807 03:08:46.372232 19581 leader_election.cc:302] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19aed34633f6458f899019e9f54fcdc9, 6341165f95f14c2bb7df240e548bc06e, 81fb0785479c4ffe99953381648d2e97; no voters:
I20260807 03:08:46.372535 19948 raft_consensus.cc:2781] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:08:46.372674 19948 raft_consensus.cc:482] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:08:46.372797 19948 raft_consensus.cc:3037] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:46.375598 19948 raft_consensus.cc:504] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:46.376227 19948 leader_election.cc:288] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [CANDIDATE]: Term 1 election: Requested vote from peers 19aed34633f6458f899019e9f54fcdc9 (127.17.31.1:44901), 81fb0785479c4ffe99953381648d2e97 (127.17.31.3:39065)
I20260807 03:08:46.376653 19503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50d9c2e203f40a5b2988674e573c09e" candidate_uuid: "6341165f95f14c2bb7df240e548bc06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19aed34633f6458f899019e9f54fcdc9"
I20260807 03:08:46.376818 19798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50d9c2e203f40a5b2988674e573c09e" candidate_uuid: "6341165f95f14c2bb7df240e548bc06e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81fb0785479c4ffe99953381648d2e97"
I20260807 03:08:46.376914 19503 raft_consensus.cc:3037] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:46.377068 19798 raft_consensus.cc:3037] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:46.380093 19798 raft_consensus.cc:2454] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6341165f95f14c2bb7df240e548bc06e in term 1.
I20260807 03:08:46.380208 19503 raft_consensus.cc:2454] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6341165f95f14c2bb7df240e548bc06e in term 1.
I20260807 03:08:46.380641 19578 leader_election.cc:302] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6341165f95f14c2bb7df240e548bc06e, 81fb0785479c4ffe99953381648d2e97; no voters:
I20260807 03:08:46.381017 19948 raft_consensus.cc:2781] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:08:46.381649 19948 raft_consensus.cc:686] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [term 1 LEADER]: Becoming Leader. State: Replica: 6341165f95f14c2bb7df240e548bc06e, State: Running, Role: LEADER
I20260807 03:08:46.382050 19948 consensus_queue.cc:237] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } }
I20260807 03:08:46.385267 19337 catalog_manager.cc:5495] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6341165f95f14c2bb7df240e548bc06e (127.17.31.2). New cstate: current_term: 1 leader_uuid: "6341165f95f14c2bb7df240e548bc06e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6341165f95f14c2bb7df240e548bc06e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 } health_report { overall_health: UNKNOWN } } }
I20260807 03:08:46.420074 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:46.413852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34222 (local address 127.17.31.2:43991)
0807 03:08:46.414652 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:46.414656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:46.414678 (+ 22us) server_negotiation.cc:407] Connection header received
0807 03:08:46.414726 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:46.414730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:46.414785 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:46.414864 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:08:46.415364 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:46.416409 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:46.418064 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:46.418708 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:46.418800 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:46.419371 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:08:46.419486 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:08:46.419561 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:46.419690 (+ 129us) server_negotiation.cc:299] Negotiation successful
0807 03:08:46.419747 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":539,"threads_started":1}
I20260807 03:08:46.429209 19798 raft_consensus.cc:1261] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Refusing update from remote peer 6341165f95f14c2bb7df240e548bc06e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:08:46.429528 19503 raft_consensus.cc:1261] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9 [term 1 FOLLOWER]: Refusing update from remote peer 6341165f95f14c2bb7df240e548bc06e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:08:46.429862 19987 consensus_queue.cc:1035] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "81fb0785479c4ffe99953381648d2e97" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 39065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:08:46.430382 19948 consensus_queue.cc:1035] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [LEADER]: Connected to new peer: Peer: permanent_uuid: "19aed34633f6458f899019e9f54fcdc9" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 03:08:46.451315 19940 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 03:08:48.024592 19336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:08:48.024905 19336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:08:48.030684 19336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a2644f81bfa4181b1ec92e82b40ec97: Sending DeleteTablet for 3 replicas of tablet a50d9c2e203f40a5b2988674e573c09e
I20260807 03:08:48.031563 19481 tablet_service.cc:1514] Processing DeleteTablet for tablet a50d9c2e203f40a5b2988674e573c09e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:48 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:37482
I20260807 03:08:48.031589 19627 tablet_service.cc:1514] Processing DeleteTablet for tablet a50d9c2e203f40a5b2988674e573c09e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:48 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:34206
I20260807 03:08:48.032174 19772 tablet_service.cc:1514] Processing DeleteTablet for tablet a50d9c2e203f40a5b2988674e573c09e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:08:48 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:50320
I20260807 03:08:48.032438 20018 tablet_replica.cc:330] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9: stopping tablet replica
I20260807 03:08:48.032693 20019 tablet_replica.cc:330] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e: stopping tablet replica
I20260807 03:08:48.032684 20018 raft_consensus.cc:2229] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:08:48.032972 20018 raft_consensus.cc:2258] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:48.033149 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 19410
I20260807 03:08:48.033504 20019 raft_consensus.cc:2229] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:08:48.034162 20019 raft_consensus.cc:2258] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:48.034400 20018 ts_tablet_manager.cc:1905] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:48.035481 20019 ts_tablet_manager.cc:1905] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:48.042834 20020 tablet_replica.cc:330] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97: stopping tablet replica
I20260807 03:08:48.044166 20018 ts_tablet_manager.cc:1918] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 03:08:48.044453 20018 log.cc:1196] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/a50d9c2e203f40a5b2988674e573c09e
I20260807 03:08:48.044582 20020 raft_consensus.cc:2229] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:08:48.044880 20020 raft_consensus.cc:2258] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:08:48.045337 20019 ts_tablet_manager.cc:1918] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 03:08:48.045084 20018 ts_tablet_manager.cc:1939] T a50d9c2e203f40a5b2988674e573c09e P 19aed34633f6458f899019e9f54fcdc9: Deleting consensus metadata
I20260807 03:08:48.045536 20019 log.cc:1196] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/a50d9c2e203f40a5b2988674e573c09e
I20260807 03:08:48.046072 20019 ts_tablet_manager.cc:1939] T a50d9c2e203f40a5b2988674e573c09e P 6341165f95f14c2bb7df240e548bc06e: Deleting consensus metadata
I20260807 03:08:48.046624 19324 catalog_manager.cc:4841] TS 19aed34633f6458f899019e9f54fcdc9 (127.17.31.1:44901): tablet a50d9c2e203f40a5b2988674e573c09e (table test-table [id=65a07191dbec46c1a2de07b95a203773]) successfully deleted
I20260807 03:08:48.047160 20020 ts_tablet_manager.cc:1905] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:08:48.047132 19322 catalog_manager.cc:4841] TS 6341165f95f14c2bb7df240e548bc06e (127.17.31.2:43991): tablet a50d9c2e203f40a5b2988674e573c09e (table test-table [id=65a07191dbec46c1a2de07b95a203773]) successfully deleted
I20260807 03:08:48.053604 20020 ts_tablet_manager.cc:1918] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 03:08:48.053807 20020 log.cc:1196] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/a50d9c2e203f40a5b2988674e573c09e
I20260807 03:08:48.054426 20020 ts_tablet_manager.cc:1939] T a50d9c2e203f40a5b2988674e573c09e P 81fb0785479c4ffe99953381648d2e97: Deleting consensus metadata
I20260807 03:08:48.055315 19321 catalog_manager.cc:4841] TS 81fb0785479c4ffe99953381648d2e97 (127.17.31.3:39065): tablet a50d9c2e203f40a5b2988674e573c09e (table test-table [id=65a07191dbec46c1a2de07b95a203773]) successfully deleted
Aug 07 03:08:48 dist-test-slave-tm1g krb5kdc[19292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072125, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:08:48.067492 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:48.035188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.17.31.1:44901)
0807 03:08:48.044649 (+ 9461us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:48.044655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:48.044675 (+ 20us) server_negotiation.cc:407] Connection header received
0807 03:08:48.044730 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:48.044735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:48.044868 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:48.044980 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:48.047225 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:48.048060 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:48.050243 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:48.050730 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:48.050841 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:48.061611 (+ 10770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:48.061626 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:48.061644 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:48.061675 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:48.064680 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:48.065433 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:48.065441 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:48.065443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:48.065612 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:48.065974 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:48.065977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:48.065979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:48.066441 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:48.066741 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:48.066943 (+ 202us) server_negotiation.cc:299] Negotiation successful
0807 03:08:48.067141 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9282,"thread_start_us":9085,"threads_started":1}
W20260807 03:08:48.068363 19550 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37508
W20260807 03:08:48.069115 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 19410: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:08:48.069377 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 19410
I20260807 03:08:48.093689 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 19560
Aug 07 03:08:48 dist-test-slave-tm1g krb5kdc[19292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072125, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:08:48.108716 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:48.094472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.17.31.2:43991)
0807 03:08:48.095247 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:48.095261 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:48.095278 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:08:48.095332 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:48.095336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:48.095395 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:48.095491 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:48.096688 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:48.097477 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:48.098792 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:48.099176 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:48.099270 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:48.103574 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:48.103587 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:48.103603 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:48.103635 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:48.106243 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:48.106985 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:48.106991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:48.106994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:48.107105 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:48.107442 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:48.107445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:48.107447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:48.107782 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:48.108020 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:48.108291 (+ 271us) server_negotiation.cc:299] Negotiation successful
0807 03:08:48.108445 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":435,"threads_started":1}
W20260807 03:08:48.109378 19694 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34232
W20260807 03:08:48.110088 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 19560: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:08:48.110209 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 19560
I20260807 03:08:48.134470 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 19704
Aug 07 03:08:48 dist-test-slave-tm1g krb5kdc[19292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072125, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:08:48.149338 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:48.135110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.17.31.3:39065)
0807 03:08:48.135665 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:48.135670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:48.135685 (+ 15us) server_negotiation.cc:407] Connection header received
0807 03:08:48.135734 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:48.135738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:48.135797 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:48.135888 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:48.137179 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:48.137919 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:48.139164 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:48.139583 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:48.139673 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:48.143827 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:48.143838 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:48.143854 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:48.143882 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:48.146742 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:48.147413 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:48.147422 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:48.147424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:48.147550 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:48.147908 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:48.147915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:48.147921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:48.148234 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:48.148583 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:48.148793 (+ 210us) server_negotiation.cc:299] Negotiation successful
0807 03:08:48.149002 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":263,"threads_started":1}
W20260807 03:08:48.150009 19841 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50340
W20260807 03:08:48.150673 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 19704: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:08:48.150823 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 19704
I20260807 03:08:48.176584 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 19305
I20260807 03:08:48.177692 19377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:08:48.294978 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 19305
2026-08-07T03:08:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8551 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:08:48 dist-test-slave-tm1g krb5kdc[20028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:08:48 dist-test-slave-tm1g krb5kdc[20028](info): set up 2 sockets
Aug 07 03:08:48 dist-test-slave-tm1g krb5kdc[20028](info): commencing operation
krb5kdc: starting...
W20260807 03:08:49.399852 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:08:49 dist-test-slave-tm1g krb5kdc[20028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:08:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:08:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:08:49.580967 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:42531
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:42531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:49.902762 20042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:49.903046 20042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:49.903208 20042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:49.912976 20042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:08:49.913105 20042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:49.913208 20042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:49.913335 20042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:08:49.913414 20042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:08:49.923226 20042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:42531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:42531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:49.924566 20042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:49.925902 20042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:49.938657 20048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:49.938735 20047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:49.938757 20050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:49.939581 20042 server_base.cc:1034] running on GCE node
I20260807 03:08:49.940387 20042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:49.941730 20042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:49.942951 20042 hybrid_clock.cc:648] HybridClock initialized: now 1786072129942926 us; error 66 us; skew 500 ppm
I20260807 03:08:49.943290 20042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:49 dist-test-slave-tm1g krb5kdc[20028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072129, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:49.946223 20042 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:08:49.947615 20042 webserver.cc:460] Webserver started at http://127.17.31.62:34983/ using document root <none> and password file <none>
I20260807 03:08:49.948289 20042 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:49.948457 20042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:49.948761 20042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:49.951747 20042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "0179e015929f493aa608289deb994218"
format_stamp: "Formatted at 2026-08-07 03:08:49 on dist-test-slave-tm1g"
server_key: "4cb1bd8a8bfac6cb0ba8837e05ee84a0"
server_key_iv: "6936dc1f7119892fe752ff155a311821"
server_key_version: "encryptionkey@0"
I20260807 03:08:49.952492 20042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "0179e015929f493aa608289deb994218"
format_stamp: "Formatted at 2026-08-07 03:08:49 on dist-test-slave-tm1g"
server_key: "4cb1bd8a8bfac6cb0ba8837e05ee84a0"
server_key_iv: "6936dc1f7119892fe752ff155a311821"
server_key_version: "encryptionkey@0"
I20260807 03:08:49.957626 20042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 03:08:49.960824 20057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:49.962030 20042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 03:08:49.962208 20042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "0179e015929f493aa608289deb994218"
format_stamp: "Formatted at 2026-08-07 03:08:49 on dist-test-slave-tm1g"
server_key: "4cb1bd8a8bfac6cb0ba8837e05ee84a0"
server_key_iv: "6936dc1f7119892fe752ff155a311821"
server_key_version: "encryptionkey@0"
I20260807 03:08:49.962484 20042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:49.978446 20042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:49.982075 20042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:49.982404 20042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:50.000913 20042 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:42531
I20260807 03:08:50.000953 20109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:42531 every 8 connection(s)
I20260807 03:08:50.002223 20042 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:08:50.005731 20110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:50.007205 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 20042
I20260807 03:08:50.007447 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:08:50.008263 17532 external_mini_cluster.cc:1428] Setting key 669b97a0a1d0ece12182a9542fc4ae8a
I20260807 03:08:50.019554 20110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218: Bootstrap starting.
Aug 07 03:08:50 dist-test-slave-tm1g krb5kdc[20028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:50.023274 20110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:50.024283 20110 log.cc:824] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218: Log is configured to *not* fsync() on all Append() calls
I20260807 03:08:50.027603 20110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218: No bootstrap required, opened a new log
I20260807 03:08:50.032030 20113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:50.010273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.17.31.62:42531)
0807 03:08:50.011223 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:50.011238 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:50.011277 (+ 39us) server_negotiation.cc:407] Connection header received
0807 03:08:50.012849 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:50.012873 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:50.013230 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:50.013526 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:50.015277 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:50.016509 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:50.018418 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:50.019115 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:50.019243 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:50.023928 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:50.023935 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:50.023949 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:50.023983 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:50.027719 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:50.028381 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:50.028389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:50.028396 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:50.028589 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:50.028957 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:50.028964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:50.028970 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:50.029627 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:50.029947 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:50.030237 (+ 290us) server_negotiation.cc:299] Negotiation successful
0807 03:08:50.030687 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":490,"threads_started":1}
I20260807 03:08:50.036798 20110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0179e015929f493aa608289deb994218" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42531 } }
I20260807 03:08:50.037133 20110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:50.037195 20110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0179e015929f493aa608289deb994218, State: Initialized, Role: FOLLOWER
I20260807 03:08:50.037698 20110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0179e015929f493aa608289deb994218" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42531 } }
I20260807 03:08:50.037892 20110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:08:50.037988 20110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:08:50.038100 20110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:50.040096 20110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0179e015929f493aa608289deb994218" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42531 } }
I20260807 03:08:50.040649 20110 leader_election.cc:302] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0179e015929f493aa608289deb994218; no voters:
I20260807 03:08:50.041298 20110 leader_election.cc:288] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:08:50.041424 20115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:08:50.042109 20115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [term 1 LEADER]: Becoming Leader. State: Replica: 0179e015929f493aa608289deb994218, State: Running, Role: LEADER
I20260807 03:08:50.042590 20115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0179e015929f493aa608289deb994218" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42531 } }
I20260807 03:08:50.043241 20110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:08:50.045532 20116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0179e015929f493aa608289deb994218" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0179e015929f493aa608289deb994218" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42531 } } }
I20260807 03:08:50.046036 20116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [sys.catalog]: This master's current role is: LEADER
I20260807 03:08:50.045644 20117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0179e015929f493aa608289deb994218. Latest consensus state: current_term: 1 leader_uuid: "0179e015929f493aa608289deb994218" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0179e015929f493aa608289deb994218" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42531 } } }
I20260807 03:08:50.046386 20117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218 [sys.catalog]: This master's current role is: LEADER
I20260807 03:08:50.048141 20123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:08:50.053332 20123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:08:50.062333 20123 catalog_manager.cc:1347] Generated new cluster ID: 5a888cdc5f11406fac5bbc42016939c0
I20260807 03:08:50.062438 20123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:08:50.070057 20123 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:08:50.071437 20123 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:08:50.078670 20123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0179e015929f493aa608289deb994218: Generated new TSK 0
I20260807 03:08:50.079442 20123 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:08:50.126220 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42531
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:50.455672 20136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:50.455943 20136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:50.456099 20136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:50.463234 20136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:50.463340 20136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:50.463513 20136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:08:50.473237 20136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:50.474603 20136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:50.475952 20136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:50.488986 20141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:50.488963 20142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:50.489635 20144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:50.489938 20136 server_base.cc:1034] running on GCE node
I20260807 03:08:50.490593 20136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:50.491443 20136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:50.492645 20136 hybrid_clock.cc:648] HybridClock initialized: now 1786072130492581 us; error 52 us; skew 500 ppm
I20260807 03:08:50.492971 20136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:50 dist-test-slave-tm1g krb5kdc[20028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072130, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:50.496181 20136 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:08:50.497591 20136 webserver.cc:460] Webserver started at http://127.17.31.1:40431/ using document root <none> and password file <none>
I20260807 03:08:50.498286 20136 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:50.498487 20136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:50.498814 20136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:50.501713 20136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "9f218fa1649b4d56bd416458abcab627"
format_stamp: "Formatted at 2026-08-07 03:08:50 on dist-test-slave-tm1g"
server_key: "872eeabe0747f2225e8d62c59feebf18"
server_key_iv: "590f04e120d927d3b529c5f5124d5594"
server_key_version: "encryptionkey@0"
I20260807 03:08:50.502586 20136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "9f218fa1649b4d56bd416458abcab627"
format_stamp: "Formatted at 2026-08-07 03:08:50 on dist-test-slave-tm1g"
server_key: "872eeabe0747f2225e8d62c59feebf18"
server_key_iv: "590f04e120d927d3b529c5f5124d5594"
server_key_version: "encryptionkey@0"
I20260807 03:08:50.507782 20136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 03:08:50.511148 20151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:50.512492 20136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:08:50.512684 20136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "9f218fa1649b4d56bd416458abcab627"
format_stamp: "Formatted at 2026-08-07 03:08:50 on dist-test-slave-tm1g"
server_key: "872eeabe0747f2225e8d62c59feebf18"
server_key_iv: "590f04e120d927d3b529c5f5124d5594"
server_key_version: "encryptionkey@0"
I20260807 03:08:50.512887 20136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:50.527711 20136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:50.531522 20136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:50.531804 20136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:50.532708 20136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:08:50.534072 20136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:50.534173 20136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:50.534349 20136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:50.534435 20136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260807 03:08:50.567720 20136 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:40505
I20260807 03:08:50.567770 20267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:40505 every 8 connection(s)
I20260807 03:08:50.569021 20136 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:08:50.575887 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 20136
I20260807 03:08:50.576104 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:08:50.576861 17532 external_mini_cluster.cc:1428] Setting key ad04c0942d6dd80874a748efb5c49532
Aug 07 03:08:50 dist-test-slave-tm1g krb5kdc[20028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072130, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:50.590361 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:50.571587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:59677 (local address 127.17.31.62:42531)
0807 03:08:50.572022 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:50.572027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:50.573044 (+ 1017us) server_negotiation.cc:407] Connection header received
0807 03:08:50.574825 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:50.574830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:50.574897 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:50.575005 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:50.576544 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:50.577626 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:50.579574 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:50.579958 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:50.580108 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:50.583962 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:50.583967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:50.583971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:50.584002 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:50.586891 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:50.587594 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:50.587604 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:50.587606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:50.587721 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:50.588289 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:50.588297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:50.588303 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:50.588703 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:50.588871 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:50.589590 (+ 719us) server_negotiation.cc:299] Negotiation successful
0807 03:08:50.589866 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":222,"threads_started":1}
I20260807 03:08:50.590930 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:50.572170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:42531 (local address 127.17.31.1:59677)
0807 03:08:50.572868 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:50.572913 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:50.574586 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:50.575196 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:50.575204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:50.575726 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:50.576375 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:50.576393 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:50.577794 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:50.577799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:50.579384 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:50.579396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:50.580191 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:50.580198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:50.580425 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:50.581198 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:50.581221 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:50.583634 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:50.587028 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:50.587037 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:50.587052 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:50.587461 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:50.587839 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:50.587845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:50.587849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:50.588118 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:50.588919 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:50.588925 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:50.589213 (+ 288us) client_negotiation.cc:769] Sending connection context
0807 03:08:50.589430 (+ 217us) client_negotiation.cc:240] Negotiation successful
0807 03:08:50.589799 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":272,"threads_started":1}
I20260807 03:08:50.592443 20268 heartbeater.cc:347] Connected to a master server at 127.17.31.62:42531
I20260807 03:08:50.592761 20268 heartbeater.cc:464] Registering TS with master...
I20260807 03:08:50.593678 20268 heartbeater.cc:511] Master 127.17.31.62:42531 requested a full tablet report, sending...
I20260807 03:08:50.595855 20074 ts_manager.cc:194] Registered new tserver with Master: 9f218fa1649b4d56bd416458abcab627 (127.17.31.1:40505)
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
I20260807 03:08:50.597293 20074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:59677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:08:50.621783 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42531
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:50.941162 20274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:50.941417 20274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:50.941598 20274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:50.949122 20274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:50.949241 20274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:50.949427 20274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:08:50.958860 20274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:50.960065 20274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:50.961318 20274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:50.974278 20282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:50.974359 20280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:50.974397 20279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:50.976002 20274 server_base.cc:1034] running on GCE node
I20260807 03:08:50.976594 20274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:50.977427 20274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:50.978652 20274 hybrid_clock.cc:648] HybridClock initialized: now 1786072130978595 us; error 46 us; skew 500 ppm
I20260807 03:08:50.978919 20274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:50 dist-test-slave-tm1g krb5kdc[20028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072130, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:50.981809 20274 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:08:50.983232 20274 webserver.cc:460] Webserver started at http://127.17.31.2:46539/ using document root <none> and password file <none>
I20260807 03:08:50.983901 20274 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:50.984035 20274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:50.984349 20274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:50.987473 20274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "54a24b606e204338a02906f43d92c3ff"
format_stamp: "Formatted at 2026-08-07 03:08:50 on dist-test-slave-tm1g"
server_key: "ccdea1122516f22deee98710cb4329da"
server_key_iv: "953ff9ed7d631f0f13aaefc0d33d0457"
server_key_version: "encryptionkey@0"
I20260807 03:08:50.988267 20274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "54a24b606e204338a02906f43d92c3ff"
format_stamp: "Formatted at 2026-08-07 03:08:50 on dist-test-slave-tm1g"
server_key: "ccdea1122516f22deee98710cb4329da"
server_key_iv: "953ff9ed7d631f0f13aaefc0d33d0457"
server_key_version: "encryptionkey@0"
I20260807 03:08:50.993659 20274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:08:50.997053 20289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:50.998294 20274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:08:50.998550 20274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "54a24b606e204338a02906f43d92c3ff"
format_stamp: "Formatted at 2026-08-07 03:08:50 on dist-test-slave-tm1g"
server_key: "ccdea1122516f22deee98710cb4329da"
server_key_iv: "953ff9ed7d631f0f13aaefc0d33d0457"
server_key_version: "encryptionkey@0"
I20260807 03:08:50.998746 20274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:51.019016 20274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:51.022729 20274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:51.023013 20274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:51.023933 20274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:08:51.025161 20274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:51.025233 20274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:51.025417 20274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:51.025468 20274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:51.058569 20274 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:39105
I20260807 03:08:51.058668 20405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:39105 every 8 connection(s)
I20260807 03:08:51.059824 20274 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:08:51.060168 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 20274
I20260807 03:08:51.060393 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:08:51.061013 17532 external_mini_cluster.cc:1428] Setting key e6f48b380f3cd807c4c3ad3ae16903f0
Aug 07 03:08:51 dist-test-slave-tm1g krb5kdc[20028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072130, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:51.082760 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:51.062503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:60663 (local address 127.17.31.62:42531)
0807 03:08:51.062672 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:51.062677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:51.064085 (+ 1408us) server_negotiation.cc:407] Connection header received
0807 03:08:51.067011 (+ 2926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:51.067015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:51.067082 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:51.067168 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:51.069218 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:51.069937 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:51.071729 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:51.072150 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:51.072292 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:51.075716 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:51.075725 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:51.075732 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:51.075778 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:51.079542 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:51.080266 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:51.080275 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:51.080282 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:51.080452 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:51.080868 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:51.080874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:51.080879 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:51.081259 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:51.081415 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:51.082074 (+ 659us) server_negotiation.cc:299] Negotiation successful
0807 03:08:51.082335 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
I20260807 03:08:51.083473 20409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:51.063063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:42531 (local address 127.17.31.2:60663)
0807 03:08:51.063893 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:51.063951 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:51.066726 (+ 2775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:51.067443 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:51.067456 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:51.068247 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:51.069038 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:51.069064 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:51.070103 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:51.070111 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:51.071577 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:51.071590 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:51.072398 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:51.072406 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:51.072633 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:51.073316 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:51.073341 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:51.075483 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:51.079682 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:51.079689 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:51.079702 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:51.080125 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:51.080578 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:51.080581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:51.080583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:51.080747 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:51.081384 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:51.081390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:51.081675 (+ 285us) client_negotiation.cc:769] Sending connection context
0807 03:08:51.081903 (+ 228us) client_negotiation.cc:240] Negotiation successful
0807 03:08:51.082261 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":326,"threads_started":1}
I20260807 03:08:51.084973 20406 heartbeater.cc:347] Connected to a master server at 127.17.31.62:42531
I20260807 03:08:51.085300 20406 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
I20260807 03:08:51.086148 20406 heartbeater.cc:511] Master 127.17.31.62:42531 requested a full tablet report, sending...
I20260807 03:08:51.087638 20074 ts_manager.cc:194] Registered new tserver with Master: 54a24b606e204338a02906f43d92c3ff (127.17.31.2:39105)
I20260807 03:08:51.088411 20074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:60663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:08:51.107151 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42531
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:51.426599 20412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:51.426849 20412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:51.427011 20412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:51.434880 20412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:51.434973 20412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:51.435150 20412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:08:51.444545 20412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:51.445787 20412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:51.447115 20412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:51.460080 20418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:51.460162 20417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:51.460119 20420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:51.460922 20412 server_base.cc:1034] running on GCE node
I20260807 03:08:51.461493 20412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:51.462420 20412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:51.463639 20412 hybrid_clock.cc:648] HybridClock initialized: now 1786072131463589 us; error 38 us; skew 500 ppm
I20260807 03:08:51.463948 20412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:51 dist-test-slave-tm1g krb5kdc[20028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072131, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:51.467052 20412 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:08:51.468381 20412 webserver.cc:460] Webserver started at http://127.17.31.3:43581/ using document root <none> and password file <none>
I20260807 03:08:51.468987 20412 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:51.469084 20412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:51.469337 20412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:51.472118 20412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "0ef0a6998c0e43958e76c8d660e9af78"
format_stamp: "Formatted at 2026-08-07 03:08:51 on dist-test-slave-tm1g"
server_key: "0fbcbd584db729095c047d6c4208d9be"
server_key_iv: "8ffd0a391ab28244d4ea36a567514b6c"
server_key_version: "encryptionkey@0"
I20260807 03:08:51.472836 20412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "0ef0a6998c0e43958e76c8d660e9af78"
format_stamp: "Formatted at 2026-08-07 03:08:51 on dist-test-slave-tm1g"
server_key: "0fbcbd584db729095c047d6c4208d9be"
server_key_iv: "8ffd0a391ab28244d4ea36a567514b6c"
server_key_version: "encryptionkey@0"
I20260807 03:08:51.477640 20412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 03:08:51.481227 20427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:51.482548 20412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:08:51.482676 20412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "0ef0a6998c0e43958e76c8d660e9af78"
format_stamp: "Formatted at 2026-08-07 03:08:51 on dist-test-slave-tm1g"
server_key: "0fbcbd584db729095c047d6c4208d9be"
server_key_iv: "8ffd0a391ab28244d4ea36a567514b6c"
server_key_version: "encryptionkey@0"
I20260807 03:08:51.482825 20412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:51.494148 20412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:51.497874 20412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:51.498098 20412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:51.499063 20412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:08:51.500255 20412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:51.500334 20412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:51.500422 20412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:51.500465 20412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:51.533151 20412 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:36687
I20260807 03:08:51.533236 20542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:36687 every 8 connection(s)
I20260807 03:08:51.534505 20412 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:08:51.544610 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 20412
I20260807 03:08:51.544823 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:08:51.545253 17532 external_mini_cluster.cc:1428] Setting key 25969772679d0323762e57466822f394
Aug 07 03:08:51 dist-test-slave-tm1g krb5kdc[20028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072131, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:51.553697 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:51.536977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:59833 (local address 127.17.31.62:42531)
0807 03:08:51.537151 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:51.537155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:51.538357 (+ 1202us) server_negotiation.cc:407] Connection header received
0807 03:08:51.540127 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:51.540134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:51.540250 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:51.540365 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:51.541812 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:51.542651 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:51.544231 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:51.544751 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:51.544883 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:51.548790 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:51.548795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:51.548798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:51.548836 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:51.551198 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:51.551837 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:51.551841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:51.551843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:51.551961 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:51.552312 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:51.552316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:51.552318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:51.552575 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:51.552684 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:51.553296 (+ 612us) server_negotiation.cc:299] Negotiation successful
0807 03:08:51.553461 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 03:08:51.554697 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:51.537457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:42531 (local address 127.17.31.3:59833)
0807 03:08:51.538139 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:51.538229 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:51.539850 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:51.540601 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:51.540609 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:51.541105 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:51.541644 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:51.541660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:51.542807 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:51.542811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:51.544075 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:51.544087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:51.544973 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:51.544982 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:51.545225 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:51.545901 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:51.545922 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:51.548612 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:51.551322 (+ 2710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:51.551328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:51.551339 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:51.551731 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:51.552055 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:51.552058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:51.552060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:51.552222 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:51.552677 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:51.552682 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:51.552964 (+ 282us) client_negotiation.cc:769] Sending connection context
0807 03:08:51.553198 (+ 234us) client_negotiation.cc:240] Negotiation successful
0807 03:08:51.553552 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":269,"threads_started":1}
I20260807 03:08:51.555963 20543 heartbeater.cc:347] Connected to a master server at 127.17.31.62:42531
I20260807 03:08:51.556222 20543 heartbeater.cc:464] Registering TS with master...
I20260807 03:08:51.556833 20543 heartbeater.cc:511] Master 127.17.31.62:42531 requested a full tablet report, sending...
I20260807 03:08:51.558118 20074 ts_manager.cc:194] Registered new tserver with Master: 0ef0a6998c0e43958e76c8d660e9af78 (127.17.31.3:36687)
I20260807 03:08:51.559139 20074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:59833
I20260807 03:08:51.560863 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 03:08:51.599766 20270 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:08:51.597269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33466 (local address 127.17.31.62:42531)
0807 03:08:51.597470 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:51.597479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:51.597911 (+ 432us) server_negotiation.cc:407] Connection header received
0807 03:08:51.598135 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:51.598139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:51.598202 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:51.598312 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:51.599569 (+ 1257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33466: BlockingRecv error: recv got EOF from 127.0.0.1:33466 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 03:08:51.600207 20268 heartbeater.cc:503] Master 127.17.31.62:42531 was elected leader, sending a full tablet report...
I20260807 03:08:51.603751 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 20136
W20260807 03:08:51.606190 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 20136: Not authorized: Client connection negotiation failed: client connection to 127.17.31.1:40505: server requires authentication, but client does not have Kerberos credentials available
I20260807 03:08:51.606387 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 20136
I20260807 03:08:51.622754 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 20274
W20260807 03:08:51.625325 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 20274: Not authorized: Client connection negotiation failed: client connection to 127.17.31.2:39105: server requires authentication, but client does not have Kerberos credentials available
I20260807 03:08:51.625495 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 20274
I20260807 03:08:51.641237 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 20412
W20260807 03:08:51.644207 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 20412: Not authorized: Client connection negotiation failed: client connection to 127.17.31.3:36687: server requires authentication, but client does not have Kerberos credentials available
I20260807 03:08:51.644315 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 20412
I20260807 03:08:51.660919 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 20042
I20260807 03:08:51.662048 20105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:08:51.766716 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 20042
2026-08-07T03:08:51Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3469 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:08:51 dist-test-slave-tm1g krb5kdc[20560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:08:51 dist-test-slave-tm1g krb5kdc[20560](info): set up 2 sockets
Aug 07 03:08:51 dist-test-slave-tm1g krb5kdc[20560](info): commencing operation
krb5kdc: starting...
W20260807 03:08:52.920588 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260807 03:08:53.081636 17532 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.161s user 0.001s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260807 03:08:53.233000 17532 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.151s user 0.003s sys 0.003s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260807 03:08:53.401186 17532 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.168s user 0.000s sys 0.007s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/krb5kdc/test-user.keytab.
W20260807 03:08:53.967422 17532 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.566s user 0.000s sys 0.006s
Aug 07 03:08:53 dist-test-slave-tm1g krb5kdc[20560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:08:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:08:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
W20260807 03:08:54.416626 17532 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.17.31.62: real 0.396s user 0.001s sys 0.009s
I20260807 03:08:54.417716 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:40583
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:44991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:40583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:54.738402 20573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:54.738688 20573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:54.738866 20573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:54.745914 20573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:08:54.746009 20573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:54.746119 20573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:54.746222 20573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:08:54.746322 20573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:08:54.755852 20573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:44991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:40583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:40583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:54.757128 20573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:54.758463 20573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:54.770231 20578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:54.770259 20581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:54.770323 20579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:54.771075 20573 server_base.cc:1034] running on GCE node
I20260807 03:08:54.771790 20573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:54.773068 20573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:54.774329 20573 hybrid_clock.cc:648] HybridClock initialized: now 1786072134774265 us; error 48 us; skew 500 ppm
I20260807 03:08:54.774667 20573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:54 dist-test-slave-tm1g krb5kdc[20560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072134, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:54.782374 20573 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:08:54.783742 20573 webserver.cc:460] Webserver started at http://127.17.31.62:34507/ using document root <none> and password file <none>
I20260807 03:08:54.784422 20573 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:54.784567 20573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:54.784883 20573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:54.787860 20573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "d8f11d4763b849ed873fe6519af45b19"
format_stamp: "Formatted at 2026-08-07 03:08:54 on dist-test-slave-tm1g"
server_key: "f83aba02919b2eaabd601504158d83ed"
server_key_iv: "30d8f46fc21ce3ae92b03d2304f0407c"
server_key_version: "encryptionkey@0"
I20260807 03:08:54.788657 20573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "d8f11d4763b849ed873fe6519af45b19"
format_stamp: "Formatted at 2026-08-07 03:08:54 on dist-test-slave-tm1g"
server_key: "f83aba02919b2eaabd601504158d83ed"
server_key_iv: "30d8f46fc21ce3ae92b03d2304f0407c"
server_key_version: "encryptionkey@0"
I20260807 03:08:54.793687 20573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 03:08:54.796942 20588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:54.798203 20573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:08:54.798408 20573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "d8f11d4763b849ed873fe6519af45b19"
format_stamp: "Formatted at 2026-08-07 03:08:54 on dist-test-slave-tm1g"
server_key: "f83aba02919b2eaabd601504158d83ed"
server_key_iv: "30d8f46fc21ce3ae92b03d2304f0407c"
server_key_version: "encryptionkey@0"
I20260807 03:08:54.798596 20573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:54.824997 20573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:54.829272 20573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:54.829571 20573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:54.848131 20573 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:40583
I20260807 03:08:54.848166 20639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:40583 every 8 connection(s)
I20260807 03:08:54.849516 20573 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:08:54.853119 20640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:54.856912 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 20573
I20260807 03:08:54.857167 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:08:54.857807 17532 external_mini_cluster.cc:1428] Setting key d2109028bbb10480974a3f2e3fa7a9c7
I20260807 03:08:54.864039 20640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19: Bootstrap starting.
I20260807 03:08:54.867594 20640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:54.868729 20640 log.cc:824] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19: Log is configured to *not* fsync() on all Append() calls
I20260807 03:08:54.871999 20640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19: No bootstrap required, opened a new log
Aug 07 03:08:54 dist-test-slave-tm1g krb5kdc[20560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:54.879400 20640 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f11d4763b849ed873fe6519af45b19" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40583 } }
I20260807 03:08:54.879820 20640 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:54.879985 20640 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f11d4763b849ed873fe6519af45b19, State: Initialized, Role: FOLLOWER
I20260807 03:08:54.880620 20640 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f11d4763b849ed873fe6519af45b19" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40583 } }
I20260807 03:08:54.880870 20640 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:08:54.881037 20640 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:08:54.881060 20643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:54.859596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.17.31.62:40583)
0807 03:08:54.860459 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:54.860470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:54.860513 (+ 43us) server_negotiation.cc:407] Connection header received
0807 03:08:54.862562 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:54.862583 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:54.862931 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:54.863227 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:54.865150 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:54.866552 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:54.868337 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:54.869042 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:54.869205 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:54.873738 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:54.873745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:54.873758 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:54.873798 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:54.876809 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:54.877521 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:54.877535 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:54.877542 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:54.877719 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:54.878112 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:54.878119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:54.878127 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:54.878689 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:54.878983 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:54.879421 (+ 438us) server_negotiation.cc:299] Negotiation successful
0807 03:08:54.879860 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":388,"threads_started":1}
I20260807 03:08:54.881243 20640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:54.884066 20640 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f11d4763b849ed873fe6519af45b19" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40583 } }
I20260807 03:08:54.884588 20640 leader_election.cc:302] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8f11d4763b849ed873fe6519af45b19; no voters:
I20260807 03:08:54.884956 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:08:54.885255 20640 leader_election.cc:288] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:08:54.885370 20645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:08:54.886597 20645 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [term 1 LEADER]: Becoming Leader. State: Replica: d8f11d4763b849ed873fe6519af45b19, State: Running, Role: LEADER
I20260807 03:08:54.887218 20645 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f11d4763b849ed873fe6519af45b19" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40583 } }
I20260807 03:08:54.887616 20640 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:08:54.890074 20647 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8f11d4763b849ed873fe6519af45b19. Latest consensus state: current_term: 1 leader_uuid: "d8f11d4763b849ed873fe6519af45b19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f11d4763b849ed873fe6519af45b19" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40583 } } }
I20260807 03:08:54.890167 20646 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8f11d4763b849ed873fe6519af45b19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f11d4763b849ed873fe6519af45b19" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40583 } } }
I20260807 03:08:54.890492 20647 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [sys.catalog]: This master's current role is: LEADER
I20260807 03:08:54.890561 20646 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19 [sys.catalog]: This master's current role is: LEADER
I20260807 03:08:54.891430 20650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:08:54.905877 20650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:08:54.917054 20650 catalog_manager.cc:1347] Generated new cluster ID: f7835fba91f04077bf041ae0ec81b058
I20260807 03:08:54.917183 20650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:08:54.962428 20650 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:08:54.964367 20650 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:08:54.969228 20643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:54.952600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39768 (local address 127.17.31.62:40583)
0807 03:08:54.952848 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:54.952858 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:54.959904 (+ 7046us) server_negotiation.cc:407] Connection header received
0807 03:08:54.960405 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:54.960410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:54.960524 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:54.960605 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:54.962123 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:54.963302 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:54.965128 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:54.965827 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:54.965918 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:54.967344 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:54.967350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:54.967355 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:08:54.968320 (+ 965us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:54.968390 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:08:54.968435 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:54.968511 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:54.968747 (+ 236us) server_negotiation.cc:299] Negotiation successful
0807 03:08:54.968879 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 03:08:54.971825 20650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8f11d4763b849ed873fe6519af45b19: Generated new TSK 0
I20260807 03:08:54.972913 20650 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:08:54.982825 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 20573
I20260807 03:08:54.983940 20635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:08:55.072616 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 20573
2026-08-07T03:08:55Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (3300 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:08:55 dist-test-slave-tm1g krb5kdc[20675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:08:55 dist-test-slave-tm1g krb5kdc[20675](info): set up 2 sockets
Aug 07 03:08:55 dist-test-slave-tm1g krb5kdc[20675](info): commencing operation
krb5kdc: starting...
W20260807 03:08:56.172184 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:08:56 dist-test-slave-tm1g krb5kdc[20675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:08:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:08:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:08:56.361198 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:34771
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:34771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:56.681535 20688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:56.681851 20688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:56.682032 20688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:56.689097 20688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:08:56.689249 20688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:56.689363 20688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:56.689476 20688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:08:56.689565 20688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:08:56.699167 20688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:34771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:34771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:56.700441 20688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:56.701690 20688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:56.713090 20693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:56.713133 20696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:56.713156 20694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:56.714126 20688 server_base.cc:1034] running on GCE node
I20260807 03:08:56.714852 20688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:56.716069 20688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:56.717274 20688 hybrid_clock.cc:648] HybridClock initialized: now 1786072136717222 us; error 45 us; skew 500 ppm
I20260807 03:08:56.717585 20688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:56 dist-test-slave-tm1g krb5kdc[20675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072136, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:56.720496 20688 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:08:56.721738 20688 webserver.cc:460] Webserver started at http://127.17.31.62:33675/ using document root <none> and password file <none>
I20260807 03:08:56.722385 20688 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:56.722491 20688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:56.722721 20688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:56.725448 20688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "0357035ccd10469683c9838416a4dd57"
format_stamp: "Formatted at 2026-08-07 03:08:56 on dist-test-slave-tm1g"
server_key: "9f81f83200c31923e42294c63ac3b570"
server_key_iv: "26fb8bee7d9ad8f827edf698d1077b94"
server_key_version: "encryptionkey@0"
I20260807 03:08:56.726127 20688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "0357035ccd10469683c9838416a4dd57"
format_stamp: "Formatted at 2026-08-07 03:08:56 on dist-test-slave-tm1g"
server_key: "9f81f83200c31923e42294c63ac3b570"
server_key_iv: "26fb8bee7d9ad8f827edf698d1077b94"
server_key_version: "encryptionkey@0"
I20260807 03:08:56.731130 20688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260807 03:08:56.734210 20703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:56.735437 20688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 03:08:56.735622 20688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "0357035ccd10469683c9838416a4dd57"
format_stamp: "Formatted at 2026-08-07 03:08:56 on dist-test-slave-tm1g"
server_key: "9f81f83200c31923e42294c63ac3b570"
server_key_iv: "26fb8bee7d9ad8f827edf698d1077b94"
server_key_version: "encryptionkey@0"
I20260807 03:08:56.735810 20688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:56.769197 20688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:56.772898 20688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:56.773191 20688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:56.791260 20688 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:34771
I20260807 03:08:56.791301 20756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:34771 every 8 connection(s)
I20260807 03:08:56.792580 20688 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:08:56.796084 20757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:08:56.798178 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 20688
I20260807 03:08:56.798456 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:08:56.799103 17532 external_mini_cluster.cc:1428] Setting key b5abd2182ae93309ce08beec10e99f5a
I20260807 03:08:56.807921 20757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57: Bootstrap starting.
I20260807 03:08:56.811666 20757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57: Neither blocks nor log segments found. Creating new log.
I20260807 03:08:56.813007 20757 log.cc:824] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57: Log is configured to *not* fsync() on all Append() calls
Aug 07 03:08:56 dist-test-slave-tm1g krb5kdc[20675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:56.816118 20757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57: No bootstrap required, opened a new log
I20260807 03:08:56.821946 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:56.801046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45258 (local address 127.17.31.62:34771)
0807 03:08:56.801806 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:56.801817 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:56.801855 (+ 38us) server_negotiation.cc:407] Connection header received
0807 03:08:56.803516 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:56.803537 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:56.803878 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:56.804190 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:56.805925 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:56.807233 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:56.808979 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:56.809642 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:56.809827 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:56.814836 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:56.814843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:56.814857 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:56.814889 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:56.817876 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:56.818649 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:56.818656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:56.818661 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:56.818826 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:56.819218 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:56.819225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:56.819228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:56.819748 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:56.820120 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:56.820531 (+ 411us) server_negotiation.cc:299] Negotiation successful
0807 03:08:56.820937 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":375,"threads_started":1}
I20260807 03:08:56.824497 20757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0357035ccd10469683c9838416a4dd57" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34771 } }
I20260807 03:08:56.824949 20757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:08:56.825049 20757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0357035ccd10469683c9838416a4dd57, State: Initialized, Role: FOLLOWER
I20260807 03:08:56.825537 20757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0357035ccd10469683c9838416a4dd57" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34771 } }
I20260807 03:08:56.825735 20757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:08:56.825832 20757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:08:56.825951 20757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:08:56.828033 20757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0357035ccd10469683c9838416a4dd57" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34771 } }
I20260807 03:08:56.828445 20757 leader_election.cc:302] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0357035ccd10469683c9838416a4dd57; no voters:
I20260807 03:08:56.828969 20757 leader_election.cc:288] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:08:56.829125 20762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:08:56.829854 20762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [term 1 LEADER]: Becoming Leader. State: Replica: 0357035ccd10469683c9838416a4dd57, State: Running, Role: LEADER
I20260807 03:08:56.830454 20762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0357035ccd10469683c9838416a4dd57" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34771 } }
I20260807 03:08:56.831255 20757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:08:56.833485 20763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0357035ccd10469683c9838416a4dd57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0357035ccd10469683c9838416a4dd57" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34771 } } }
I20260807 03:08:56.833932 20763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [sys.catalog]: This master's current role is: LEADER
I20260807 03:08:56.834542 20764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0357035ccd10469683c9838416a4dd57. Latest consensus state: current_term: 1 leader_uuid: "0357035ccd10469683c9838416a4dd57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0357035ccd10469683c9838416a4dd57" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34771 } } }
I20260807 03:08:56.834993 20764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57 [sys.catalog]: This master's current role is: LEADER
I20260807 03:08:56.835768 20769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:08:56.842327 20769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:08:56.851356 20769 catalog_manager.cc:1347] Generated new cluster ID: 103cd5f8e28545f5b405c800625eddbd
I20260807 03:08:56.851454 20769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:08:56.875573 20769 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:08:56.876886 20769 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:08:56.884531 20769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0357035ccd10469683c9838416a4dd57: Generated new TSK 0
I20260807 03:08:56.885308 20769 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:08:56.947283 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:34771
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:57.281600 20783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:57.281859 20783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:57.281950 20783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:57.288997 20783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:57.289098 20783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:57.289204 20783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:08:57.298422 20783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:34771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:57.299728 20783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:57.300946 20783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:57.313786 20788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:57.313751 20789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:57.314090 20791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:57.314917 20783 server_base.cc:1034] running on GCE node
I20260807 03:08:57.315436 20783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:57.316260 20783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:57.317456 20783 hybrid_clock.cc:648] HybridClock initialized: now 1786072137317420 us; error 32 us; skew 500 ppm
I20260807 03:08:57.317761 20783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:57 dist-test-slave-tm1g krb5kdc[20675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072137, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:57.320926 20783 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:08:57.322510 20783 webserver.cc:460] Webserver started at http://127.17.31.1:43735/ using document root <none> and password file <none>
I20260807 03:08:57.323215 20783 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:57.323367 20783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:57.323673 20783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:57.326551 20783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "67a7aa6744f047c5b6d4967eca4dc9a1"
format_stamp: "Formatted at 2026-08-07 03:08:57 on dist-test-slave-tm1g"
server_key: "414576bf26f4a9f42bdc2a2328166850"
server_key_iv: "b79bf6e10e748e58dcb3732cb9b3adca"
server_key_version: "encryptionkey@0"
I20260807 03:08:57.327316 20783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "67a7aa6744f047c5b6d4967eca4dc9a1"
format_stamp: "Formatted at 2026-08-07 03:08:57 on dist-test-slave-tm1g"
server_key: "414576bf26f4a9f42bdc2a2328166850"
server_key_iv: "b79bf6e10e748e58dcb3732cb9b3adca"
server_key_version: "encryptionkey@0"
I20260807 03:08:57.332232 20783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 03:08:57.335726 20799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:57.336995 20783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:08:57.337177 20783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "67a7aa6744f047c5b6d4967eca4dc9a1"
format_stamp: "Formatted at 2026-08-07 03:08:57 on dist-test-slave-tm1g"
server_key: "414576bf26f4a9f42bdc2a2328166850"
server_key_iv: "b79bf6e10e748e58dcb3732cb9b3adca"
server_key_version: "encryptionkey@0"
I20260807 03:08:57.337380 20783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:57.349992 20783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:57.354079 20783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:57.354524 20783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:57.355436 20783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:08:57.356678 20783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:57.356750 20783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:57.356892 20783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:57.357003 20783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:57.390275 20783 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:43875
I20260807 03:08:57.390383 20913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:43875 every 8 connection(s)
I20260807 03:08:57.391602 20783 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:08:57.398572 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 20783
I20260807 03:08:57.398803 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:08:57.399636 17532 external_mini_cluster.cc:1428] Setting key 6b6f5c950cde83de01f60009023c427a
Aug 07 03:08:57 dist-test-slave-tm1g krb5kdc[20675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072137, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:57.411836 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:57.394038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:33881 (local address 127.17.31.62:34771)
0807 03:08:57.394506 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:57.394511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:57.395436 (+ 925us) server_negotiation.cc:407] Connection header received
0807 03:08:57.397228 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:57.397234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:57.397322 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:57.397455 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:57.399089 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:57.399994 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:57.401803 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:57.402202 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:57.402383 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:57.405944 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:57.405949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:57.405952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:57.405983 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:57.408681 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:57.409377 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:57.409382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:57.409386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:57.409510 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:57.409976 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:57.409987 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:57.409990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:57.410410 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:57.410577 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:57.411430 (+ 853us) server_negotiation.cc:299] Negotiation successful
0807 03:08:57.411588 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":197,"threads_started":1}
I20260807 03:08:57.413282 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:57.394588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:34771 (local address 127.17.31.1:33881)
0807 03:08:57.395269 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:57.395310 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:57.396956 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:57.397675 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:57.397683 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:57.398156 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:57.398829 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:57.398874 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:57.400155 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:57.400161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:57.401648 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:57.401660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:57.402461 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:57.402470 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:57.402706 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:57.403382 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:57.403410 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:57.405719 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:57.408828 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:57.408836 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:57.408850 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:57.409245 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:57.409628 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:57.409633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:57.409637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:57.409846 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:57.410564 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:57.410574 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:57.410990 (+ 416us) client_negotiation.cc:769] Sending connection context
0807 03:08:57.411311 (+ 321us) client_negotiation.cc:240] Negotiation successful
0807 03:08:57.411870 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":267,"threads_started":1}
I20260807 03:08:57.414808 20914 heartbeater.cc:347] Connected to a master server at 127.17.31.62:34771
I20260807 03:08:57.415115 20914 heartbeater.cc:464] Registering TS with master...
I20260807 03:08:57.415828 20914 heartbeater.cc:511] Master 127.17.31.62:34771 requested a full tablet report, sending...
I20260807 03:08:57.417918 20721 ts_manager.cc:194] Registered new tserver with Master: 67a7aa6744f047c5b6d4967eca4dc9a1 (127.17.31.1:43875)
I20260807 03:08:57.419409 20721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:33881
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:08:57.446501 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:34771
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:57.773555 20920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:57.773818 20920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:57.773989 20920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:57.781666 20920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:57.781798 20920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:57.781967 20920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:08:57.791589 20920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:34771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:57.792785 20920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:57.794101 20920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:57.807374 20926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:57.807466 20925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:57.807430 20928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:57.808205 20920 server_base.cc:1034] running on GCE node
I20260807 03:08:57.808799 20920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:57.809674 20920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:57.810901 20920 hybrid_clock.cc:648] HybridClock initialized: now 1786072137810871 us; error 47 us; skew 500 ppm
I20260807 03:08:57.811208 20920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:57 dist-test-slave-tm1g krb5kdc[20675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072137, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:57.814200 20920 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:08:57.815693 20920 webserver.cc:460] Webserver started at http://127.17.31.2:37163/ using document root <none> and password file <none>
I20260807 03:08:57.816378 20920 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:57.816530 20920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:57.816838 20920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:57.819727 20920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "70cef3c94c7d4d8eae2f90eddec35569"
format_stamp: "Formatted at 2026-08-07 03:08:57 on dist-test-slave-tm1g"
server_key: "546ee465da5e7e35bea867d73f80cc7c"
server_key_iv: "fd5d60192dbec15ebe0d8cb443a9f6d8"
server_key_version: "encryptionkey@0"
I20260807 03:08:57.820506 20920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "70cef3c94c7d4d8eae2f90eddec35569"
format_stamp: "Formatted at 2026-08-07 03:08:57 on dist-test-slave-tm1g"
server_key: "546ee465da5e7e35bea867d73f80cc7c"
server_key_iv: "fd5d60192dbec15ebe0d8cb443a9f6d8"
server_key_version: "encryptionkey@0"
I20260807 03:08:57.825673 20920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 03:08:57.829082 20935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:57.830289 20920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:08:57.830545 20920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "70cef3c94c7d4d8eae2f90eddec35569"
format_stamp: "Formatted at 2026-08-07 03:08:57 on dist-test-slave-tm1g"
server_key: "546ee465da5e7e35bea867d73f80cc7c"
server_key_iv: "fd5d60192dbec15ebe0d8cb443a9f6d8"
server_key_version: "encryptionkey@0"
I20260807 03:08:57.830736 20920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:57.861939 20920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:57.865908 20920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:57.866223 20920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:57.867228 20920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:08:57.868527 20920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:57.868638 20920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:57.868803 20920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:57.868887 20920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:57.902017 20920 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:35393
I20260807 03:08:57.902108 21051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:35393 every 8 connection(s)
I20260807 03:08:57.903386 20920 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:08:57.905095 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 20920
I20260807 03:08:57.905332 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:08:57.905838 17532 external_mini_cluster.cc:1428] Setting key 7e44ce4ff074541f94824dfd15aae656
Aug 07 03:08:57 dist-test-slave-tm1g krb5kdc[20675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072137, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:57.927314 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:57.906024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:44741 (local address 127.17.31.62:34771)
0807 03:08:57.906244 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:57.906250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:57.907608 (+ 1358us) server_negotiation.cc:407] Connection header received
0807 03:08:57.910212 (+ 2604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:57.910216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:57.910278 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:57.910434 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:57.912485 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:57.913177 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:57.915690 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:57.916422 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:57.916586 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:57.920087 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:57.920096 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:57.920104 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:57.920149 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:57.923742 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:57.924444 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:57.924452 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:57.924459 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:57.924768 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:57.925216 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:57.925225 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:57.925231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:57.925650 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:57.925815 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:57.926517 (+ 702us) server_negotiation.cc:299] Negotiation successful
0807 03:08:57.926790 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 03:08:57.927906 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:57.906661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:34771 (local address 127.17.31.2:44741)
0807 03:08:57.907449 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:57.907493 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:57.909917 (+ 2424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:57.910686 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:57.910700 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:57.911475 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:57.912291 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:57.912318 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:57.913361 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:57.913369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:57.915512 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:57.915530 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:57.916667 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:57.916672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:57.916872 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:57.917626 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:57.917648 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:57.919855 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:57.923876 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:57.923883 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:57.923895 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:57.924295 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:57.924901 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:57.924904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:57.924907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:57.925082 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:57.925787 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:57.925793 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:57.926104 (+ 311us) client_negotiation.cc:769] Sending connection context
0807 03:08:57.926399 (+ 295us) client_negotiation.cc:240] Negotiation successful
0807 03:08:57.926784 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":345,"threads_started":1}
I20260807 03:08:57.929353 21052 heartbeater.cc:347] Connected to a master server at 127.17.31.62:34771
I20260807 03:08:57.929672 21052 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
I20260807 03:08:57.930506 21052 heartbeater.cc:511] Master 127.17.31.62:34771 requested a full tablet report, sending...
I20260807 03:08:57.932085 20721 ts_manager.cc:194] Registered new tserver with Master: 70cef3c94c7d4d8eae2f90eddec35569 (127.17.31.2:35393)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
I20260807 03:08:57.933108 20721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:44741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:08:57.958294 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:34771
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:08:58.280016 21057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:08:58.280277 21057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:08:58.280383 21057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:08:58.287405 21057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:08:58.287503 21057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:08:58.287609 21057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:08:58.296931 21057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:34771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:08:58.298123 21057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:08:58.299377 21057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:08:58.312318 21062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:58.312440 21063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:08:58.312481 21065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:08:58.313488 21057 server_base.cc:1034] running on GCE node
I20260807 03:08:58.314182 21057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:08:58.315162 21057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:08:58.316437 21057 hybrid_clock.cc:648] HybridClock initialized: now 1786072138316339 us; error 74 us; skew 500 ppm
I20260807 03:08:58.316756 21057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:08:58 dist-test-slave-tm1g krb5kdc[20675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072138, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:08:58.319936 21057 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:08:58.321355 21057 webserver.cc:460] Webserver started at http://127.17.31.3:42563/ using document root <none> and password file <none>
I20260807 03:08:58.322060 21057 fs_manager.cc:359] Metadata directory not provided
I20260807 03:08:58.322220 21057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:08:58.322571 21057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:08:58.325474 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "14906581e5054e1fb50e6e3445d297da"
format_stamp: "Formatted at 2026-08-07 03:08:58 on dist-test-slave-tm1g"
server_key: "487733708cce31851efc74762a846433"
server_key_iv: "8454fe9950357380f909f9925c61ad73"
server_key_version: "encryptionkey@0"
I20260807 03:08:58.326292 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "14906581e5054e1fb50e6e3445d297da"
format_stamp: "Formatted at 2026-08-07 03:08:58 on dist-test-slave-tm1g"
server_key: "487733708cce31851efc74762a846433"
server_key_iv: "8454fe9950357380f909f9925c61ad73"
server_key_version: "encryptionkey@0"
I20260807 03:08:58.331316 21057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 03:08:58.334908 21073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:58.336218 21057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 03:08:58.336416 21057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "14906581e5054e1fb50e6e3445d297da"
format_stamp: "Formatted at 2026-08-07 03:08:58 on dist-test-slave-tm1g"
server_key: "487733708cce31851efc74762a846433"
server_key_iv: "8454fe9950357380f909f9925c61ad73"
server_key_version: "encryptionkey@0"
I20260807 03:08:58.336659 21057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:08:58.354471 21057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:08:58.358232 21057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:08:58.358564 21057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:08:58.359483 21057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:08:58.360780 21057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:08:58.360893 21057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:58.361061 21057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:08:58.361142 21057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:08:58.395769 21057 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:33603
I20260807 03:08:58.395813 21187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:33603 every 8 connection(s)
I20260807 03:08:58.396975 21057 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:08:58.397778 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 21057
I20260807 03:08:58.397981 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:08:58.398576 17532 external_mini_cluster.cc:1428] Setting key 625d195aa6e41baf34d65e5c00ae4e19
Aug 07 03:08:58 dist-test-slave-tm1g krb5kdc[20675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072138, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:58.417358 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:58.399771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:44733 (local address 127.17.31.62:34771)
0807 03:08:58.399972 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:58.399977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:58.401434 (+ 1457us) server_negotiation.cc:407] Connection header received
0807 03:08:58.403404 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:58.403409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:58.403524 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:58.403636 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:58.405028 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:58.405817 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:58.407385 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:58.407788 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:58.407929 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:58.411383 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:58.411388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:58.411392 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:58.411422 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:58.414459 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:58.415126 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:58.415131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:58.415134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:58.415265 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:58.415695 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:58.415699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:58.415703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:58.415960 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:58.416157 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:58.416794 (+ 637us) server_negotiation.cc:299] Negotiation successful
0807 03:08:58.416999 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 03:08:58.418210 21190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:58.400369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:34771 (local address 127.17.31.3:44733)
0807 03:08:58.401164 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0807 03:08:58.401234 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0807 03:08:58.403145 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:08:58.403859 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:08:58.403867 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:08:58.404339 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:08:58.404853 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:58.404869 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:58.405973 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:58.405978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:58.407227 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:08:58.407239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:08:58.408013 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:08:58.408023 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:08:58.408260 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:58.408944 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:08:58.409012 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:08:58.411176 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:08:58.414592 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:58.414601 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:58.414614 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:58.414996 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:58.415379 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:08:58.415385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:08:58.415389 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:08:58.415572 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:08:58.416128 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:08:58.416135 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:08:58.416435 (+ 300us) client_negotiation.cc:769] Sending connection context
0807 03:08:58.416696 (+ 261us) client_negotiation.cc:240] Negotiation successful
0807 03:08:58.417169 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":363,"threads_started":1}
I20260807 03:08:58.419526 21188 heartbeater.cc:347] Connected to a master server at 127.17.31.62:34771
I20260807 03:08:58.419811 21188 heartbeater.cc:464] Registering TS with master...
I20260807 03:08:58.420516 21188 heartbeater.cc:511] Master 127.17.31.62:34771 requested a full tablet report, sending...
I20260807 03:08:58.421828 20721 ts_manager.cc:194] Registered new tserver with Master: 14906581e5054e1fb50e6e3445d297da (127.17.31.3:33603)
I20260807 03:08:58.422320 20914 heartbeater.cc:503] Master 127.17.31.62:34771 was elected leader, sending a full tablet report...
I20260807 03:08:58.422865 20721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:44733
I20260807 03:08:58.426488 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 03:08:58 dist-test-slave-tm1g krb5kdc[20675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072138, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 07 03:08:58 dist-test-slave-tm1g krb5kdc[20675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072138, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:08:58.476802 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:58.462714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49404 (local address 127.17.31.62:34771)
0807 03:08:58.462971 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:58.462979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:58.463355 (+ 376us) server_negotiation.cc:407] Connection header received
0807 03:08:58.463572 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:58.463576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:58.463644 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:58.463759 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:58.465132 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:58.465897 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:58.467246 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:58.467655 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:58.467803 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:58.471661 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:58.471666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:58.471669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:58.471701 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:58.474074 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:58.474767 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:58.474774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:58.474777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:58.474908 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:58.475318 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:58.475323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:58.475328 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:58.475646 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:58.475907 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:58.476210 (+ 303us) server_negotiation.cc:299] Negotiation successful
0807 03:08:58.476438 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260807 03:08:58.477393 20720 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49404
I20260807 03:08:58.479703 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 20783
Aug 07 03:08:58 dist-test-slave-tm1g krb5kdc[20675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072138, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:08:58.494768 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:58.480703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.17.31.1:43875)
0807 03:08:58.481384 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:58.481391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:58.481414 (+ 23us) server_negotiation.cc:407] Connection header received
0807 03:08:58.481466 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:58.481472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:58.481676 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:58.481794 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:58.482945 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:58.483939 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:58.485118 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:58.485668 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:58.485761 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:58.489584 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:58.489595 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:58.489608 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:58.489642 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:58.492475 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:58.493100 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:58.493105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:58.493107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:58.493215 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:58.493524 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:58.493527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:58.493529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:58.493833 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:58.494049 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:58.494238 (+ 189us) server_negotiation.cc:299] Negotiation successful
0807 03:08:58.494491 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":369,"threads_started":1}
W20260807 03:08:58.496048 20909 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44132
W20260807 03:08:58.497113 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 20783: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:08:58.497267 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 20783
I20260807 03:08:58.514652 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 20920
Aug 07 03:08:58 dist-test-slave-tm1g krb5kdc[20675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072138, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:08:58.530043 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:58.515473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34838 (local address 127.17.31.2:35393)
0807 03:08:58.516068 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:58.516075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:58.516101 (+ 26us) server_negotiation.cc:407] Connection header received
0807 03:08:58.516167 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:58.516173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:58.516376 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:58.516509 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:58.517674 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:58.518758 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:58.519997 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:58.520511 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:58.520602 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:58.524686 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:58.524697 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:58.524710 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:58.524739 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:58.527612 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:58.528168 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:58.528175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:58.528177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:58.528290 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:58.528641 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:58.528645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:58.528648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:58.529046 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:58.529254 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:58.529549 (+ 295us) server_negotiation.cc:299] Negotiation successful
0807 03:08:58.529735 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":295,"threads_started":1}
W20260807 03:08:58.531526 21047 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34838
W20260807 03:08:58.532665 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 20920: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:08:58.532825 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 20920
I20260807 03:08:58.551339 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 21057
Aug 07 03:08:58 dist-test-slave-tm1g krb5kdc[20675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072138, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:08:58.566938 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:08:58.552344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.17.31.3:33603)
0807 03:08:58.553028 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0807 03:08:58.553035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:08:58.553062 (+ 27us) server_negotiation.cc:407] Connection header received
0807 03:08:58.553125 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:08:58.553132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:08:58.553343 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:08:58.553476 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:08:58.554818 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:58.555847 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:58.557046 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:08:58.557589 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:08:58.557685 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:08:58.561908 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:08:58.561918 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:08:58.561931 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:08:58.561961 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:08:58.564537 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:58.565109 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:58.565115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:58.565117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:58.565231 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:08:58.565542 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:08:58.565545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:08:58.565547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:08:58.565852 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:08:58.566075 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0807 03:08:58.566361 (+ 286us) server_negotiation.cc:299] Negotiation successful
0807 03:08:58.566572 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":375,"threads_started":1}
W20260807 03:08:58.568203 21183 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52472
W20260807 03:08:58.569285 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 21057: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:08:58.569414 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 21057
I20260807 03:08:58.586607 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 20688
I20260807 03:08:58.587757 20752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:08:58.691512 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 20688
2026-08-07T03:08:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3624 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:08:58 dist-test-slave-tm1g krb5kdc[21205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:08:58 dist-test-slave-tm1g krb5kdc[21205](info): set up 2 sockets
Aug 07 03:08:58 dist-test-slave-tm1g krb5kdc[21205](info): commencing operation
krb5kdc: starting...
W20260807 03:08:59.804082 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:08:59 dist-test-slave-tm1g krb5kdc[21205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:08:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:08:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:08:59.998735 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:38967
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:38967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:00.319706 21219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:00.320037 21219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:00.320209 21219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:00.327345 21219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:09:00.327440 21219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:00.327539 21219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:00.327646 21219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:09:00.327704 21219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:09:00.337221 21219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:38967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:38967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:00.338514 21219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:00.339814 21219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:00.351742 21227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:00.351747 21225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:00.351766 21224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:00.352625 21219 server_base.cc:1034] running on GCE node
I20260807 03:09:00.353380 21219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:00.354748 21219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:00.356014 21219 hybrid_clock.cc:648] HybridClock initialized: now 1786072140355958 us; error 43 us; skew 500 ppm
I20260807 03:09:00.356330 21219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:00 dist-test-slave-tm1g krb5kdc[21205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072140, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:00.366720 21219 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:09:00.368104 21219 webserver.cc:460] Webserver started at http://127.17.31.62:44211/ using document root <none> and password file <none>
I20260807 03:09:00.368770 21219 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:00.368918 21219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:00.369241 21219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:00.372131 21219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "79815169fa134a77be50a6ee4702d615"
format_stamp: "Formatted at 2026-08-07 03:09:00 on dist-test-slave-tm1g"
server_key: "0ca49144550c0dff4a876093b82ab808"
server_key_iv: "6062b1bdc757b8aa05a39f5264121e23"
server_key_version: "encryptionkey@0"
I20260807 03:09:00.372890 21219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "79815169fa134a77be50a6ee4702d615"
format_stamp: "Formatted at 2026-08-07 03:09:00 on dist-test-slave-tm1g"
server_key: "0ca49144550c0dff4a876093b82ab808"
server_key_iv: "6062b1bdc757b8aa05a39f5264121e23"
server_key_version: "encryptionkey@0"
I20260807 03:09:00.377830 21219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:09:00.381078 21234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:00.382421 21219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 03:09:00.382597 21219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "79815169fa134a77be50a6ee4702d615"
format_stamp: "Formatted at 2026-08-07 03:09:00 on dist-test-slave-tm1g"
server_key: "0ca49144550c0dff4a876093b82ab808"
server_key_iv: "6062b1bdc757b8aa05a39f5264121e23"
server_key_version: "encryptionkey@0"
I20260807 03:09:00.382792 21219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:00.405535 21219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:00.409842 21219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:00.410142 21219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:00.428705 21219 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:38967
I20260807 03:09:00.428746 21285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:38967 every 8 connection(s)
I20260807 03:09:00.430025 21219 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:09:00.433542 21287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:00.436336 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 21219
I20260807 03:09:00.436548 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:09:00.437363 17532 external_mini_cluster.cc:1428] Setting key 268ebb6e7f2627d560ad4ab992009222
I20260807 03:09:00.446161 21287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615: Bootstrap starting.
I20260807 03:09:00.449839 21287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:00.451162 21287 log.cc:824] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615: Log is configured to *not* fsync() on all Append() calls
Aug 07 03:09:00 dist-test-slave-tm1g krb5kdc[21205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:00.454140 21287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615: No bootstrap required, opened a new log
I20260807 03:09:00.460256 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:00.439503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.17.31.62:38967)
0807 03:09:00.440269 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:00.440280 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:00.440318 (+ 38us) server_negotiation.cc:407] Connection header received
0807 03:09:00.441892 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:00.441911 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:00.442284 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:00.442683 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:00.444462 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:00.445714 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:00.447480 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:00.448124 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:00.448266 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:00.453201 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:00.453208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:00.453221 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:00.453256 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:00.456376 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:00.457155 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:00.457161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:00.457166 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:00.457315 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:00.457723 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:00.457729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:00.457731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:00.458195 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:00.458513 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:00.458891 (+ 378us) server_negotiation.cc:299] Negotiation successful
0807 03:09:00.459257 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":357,"threads_started":1}
I20260807 03:09:00.463328 21287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79815169fa134a77be50a6ee4702d615" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 38967 } }
I20260807 03:09:00.463721 21287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:00.463829 21287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79815169fa134a77be50a6ee4702d615, State: Initialized, Role: FOLLOWER
I20260807 03:09:00.464416 21287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79815169fa134a77be50a6ee4702d615" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 38967 } }
I20260807 03:09:00.464679 21287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:09:00.464820 21287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:09:00.465003 21287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:00.467285 21287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79815169fa134a77be50a6ee4702d615" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 38967 } }
I20260807 03:09:00.467852 21287 leader_election.cc:302] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79815169fa134a77be50a6ee4702d615; no voters:
I20260807 03:09:00.468416 21287 leader_election.cc:288] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:09:00.468559 21292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:00.469300 21292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [term 1 LEADER]: Becoming Leader. State: Replica: 79815169fa134a77be50a6ee4702d615, State: Running, Role: LEADER
I20260807 03:09:00.469905 21292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79815169fa134a77be50a6ee4702d615" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 38967 } }
I20260807 03:09:00.470697 21287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:09:00.473228 21293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79815169fa134a77be50a6ee4702d615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79815169fa134a77be50a6ee4702d615" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 38967 } } }
I20260807 03:09:00.473677 21293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:00.473565 21294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79815169fa134a77be50a6ee4702d615. Latest consensus state: current_term: 1 leader_uuid: "79815169fa134a77be50a6ee4702d615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79815169fa134a77be50a6ee4702d615" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 38967 } } }
I20260807 03:09:00.475029 21294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615 [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:00.475193 21302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:09:00.480341 21302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:09:00.488992 21302 catalog_manager.cc:1347] Generated new cluster ID: a9aa70184a5c4cd68366cf48a9e8713a
I20260807 03:09:00.489118 21302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:09:00.501905 21302 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:09:00.503249 21302 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:09:00.508980 21302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79815169fa134a77be50a6ee4702d615: Generated new TSK 0
I20260807 03:09:00.509755 21302 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:09:00.571471 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:38967
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:00.903302 21314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:00.903578 21314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:00.903761 21314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:00.911098 21314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:00.911201 21314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:00.911429 21314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:09:00.921128 21314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:38967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:00.922451 21314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:00.923750 21314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:00.937117 21319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:00.937187 21320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:00.937155 21322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:00.938246 21314 server_base.cc:1034] running on GCE node
I20260807 03:09:00.938845 21314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:00.939677 21314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:00.940884 21314 hybrid_clock.cc:648] HybridClock initialized: now 1786072140940837 us; error 35 us; skew 500 ppm
I20260807 03:09:00.941146 21314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:00 dist-test-slave-tm1g krb5kdc[21205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072140, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:00.944329 21314 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:09:00.945802 21314 webserver.cc:460] Webserver started at http://127.17.31.1:40103/ using document root <none> and password file <none>
I20260807 03:09:00.946547 21314 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:00.946687 21314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:00.947014 21314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:00.949805 21314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "a75e0e6cc4d64fe5878b322dffb627c3"
format_stamp: "Formatted at 2026-08-07 03:09:00 on dist-test-slave-tm1g"
server_key: "ab2e56a80ba1da77a01394c7ee41053e"
server_key_iv: "15f72ee1a6e80567c830cb4c6da3f119"
server_key_version: "encryptionkey@0"
I20260807 03:09:00.950654 21314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "a75e0e6cc4d64fe5878b322dffb627c3"
format_stamp: "Formatted at 2026-08-07 03:09:00 on dist-test-slave-tm1g"
server_key: "ab2e56a80ba1da77a01394c7ee41053e"
server_key_iv: "15f72ee1a6e80567c830cb4c6da3f119"
server_key_version: "encryptionkey@0"
I20260807 03:09:00.955677 21314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 03:09:00.959038 21329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:00.960421 21314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:09:00.960605 21314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "a75e0e6cc4d64fe5878b322dffb627c3"
format_stamp: "Formatted at 2026-08-07 03:09:00 on dist-test-slave-tm1g"
server_key: "ab2e56a80ba1da77a01394c7ee41053e"
server_key_iv: "15f72ee1a6e80567c830cb4c6da3f119"
server_key_version: "encryptionkey@0"
I20260807 03:09:00.960814 21314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:00.985191 21314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:00.989230 21314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:00.989540 21314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:00.990556 21314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:00.991843 21314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:00.991961 21314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:00.992134 21314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:00.992215 21314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:01.025818 21314 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:39953
I20260807 03:09:01.025864 21443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:39953 every 8 connection(s)
I20260807 03:09:01.027210 21314 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:09:01.033293 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 21314
I20260807 03:09:01.033567 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:09:01.034283 17532 external_mini_cluster.cc:1428] Setting key 81047c82218bf05d8a39beedc46b2f14
Aug 07 03:09:01 dist-test-slave-tm1g krb5kdc[21205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072140, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:01.048133 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:01.029738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:39203 (local address 127.17.31.62:38967)
0807 03:09:01.030231 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:01.030237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:01.031318 (+ 1081us) server_negotiation.cc:407] Connection header received
0807 03:09:01.033322 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:01.033327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:01.033433 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:01.033583 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:01.035276 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:01.036092 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:01.037650 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:01.038079 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:01.038230 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:01.041915 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:01.041924 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:01.041931 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:01.041975 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:01.044802 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:01.045787 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:01.045791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:01.045793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:01.045916 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:01.046446 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:01.046450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:01.046452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:01.046706 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:01.046814 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:01.047703 (+ 889us) server_negotiation.cc:299] Negotiation successful
0807 03:09:01.047868 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":237,"threads_started":1}
I20260807 03:09:01.049737 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:01.030415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:38967 (local address 127.17.31.1:39203)
0807 03:09:01.031147 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:01.031186 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:01.032993 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:01.033819 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:01.033827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:01.034468 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:01.035088 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:01.035113 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:01.036254 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:01.036258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:01.037487 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:01.037499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:01.038348 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:01.038358 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:01.038610 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:01.039311 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:01.039332 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:01.041678 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:01.044950 (+ 3272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:01.044962 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:01.044989 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:01.045660 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:01.046046 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:01.046053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:01.046060 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:01.046325 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:01.046838 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:01.046849 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:01.047306 (+ 457us) client_negotiation.cc:769] Sending connection context
0807 03:09:01.047610 (+ 304us) client_negotiation.cc:240] Negotiation successful
0807 03:09:01.048231 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":290,"threads_started":1}
I20260807 03:09:01.051419 21444 heartbeater.cc:347] Connected to a master server at 127.17.31.62:38967
I20260807 03:09:01.051735 21444 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:01.052541 21444 heartbeater.cc:511] Master 127.17.31.62:38967 requested a full tablet report, sending...
I20260807 03:09:01.054734 21251 ts_manager.cc:194] Registered new tserver with Master: a75e0e6cc4d64fe5878b322dffb627c3 (127.17.31.1:39953)
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
I20260807 03:09:01.056248 21251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:39203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:09:01.084987 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:38967
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:01.408089 21450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:01.408347 21450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:01.408507 21450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:01.415597 21450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:01.415695 21450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:01.415853 21450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:09:01.425077 21450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:38967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:01.426456 21450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:01.427737 21450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:01.442096 21456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:01.442368 21459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:01.442407 21457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:01.443058 21450 server_base.cc:1034] running on GCE node
I20260807 03:09:01.443691 21450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:01.444752 21450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:01.445990 21450 hybrid_clock.cc:648] HybridClock initialized: now 1786072141445928 us; error 52 us; skew 500 ppm
I20260807 03:09:01.446338 21450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:01 dist-test-slave-tm1g krb5kdc[21205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072141, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:01.449730 21450 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:09:01.451265 21450 webserver.cc:460] Webserver started at http://127.17.31.2:46449/ using document root <none> and password file <none>
I20260807 03:09:01.451874 21450 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:01.451970 21450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:01.452244 21450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:01.455173 21450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "6605bfc49def4bf690b666bf0799a615"
format_stamp: "Formatted at 2026-08-07 03:09:01 on dist-test-slave-tm1g"
server_key: "361b97f05225c3878168000a0ce3179c"
server_key_iv: "1b5898145ba00d488b9dba23950848a9"
server_key_version: "encryptionkey@0"
I20260807 03:09:01.455847 21450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "6605bfc49def4bf690b666bf0799a615"
format_stamp: "Formatted at 2026-08-07 03:09:01 on dist-test-slave-tm1g"
server_key: "361b97f05225c3878168000a0ce3179c"
server_key_iv: "1b5898145ba00d488b9dba23950848a9"
server_key_version: "encryptionkey@0"
I20260807 03:09:01.460641 21450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260807 03:09:01.463898 21466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:01.465245 21450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 03:09:01.465436 21450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "6605bfc49def4bf690b666bf0799a615"
format_stamp: "Formatted at 2026-08-07 03:09:01 on dist-test-slave-tm1g"
server_key: "361b97f05225c3878168000a0ce3179c"
server_key_iv: "1b5898145ba00d488b9dba23950848a9"
server_key_version: "encryptionkey@0"
I20260807 03:09:01.465670 21450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:01.510000 21450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:01.514693 21450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:01.514937 21450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:01.515841 21450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:01.517159 21450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:01.517232 21450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:01.517314 21450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:01.517361 21450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:01.551016 21450 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:36969
I20260807 03:09:01.551137 21581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:36969 every 8 connection(s)
I20260807 03:09:01.552400 21450 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:09:01.555577 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 21450
I20260807 03:09:01.555789 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:09:01.556257 17532 external_mini_cluster.cc:1428] Setting key 1c31bdda780fe9adab422a2026c93db6
Aug 07 03:09:01 dist-test-slave-tm1g krb5kdc[21205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072141, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:01.574719 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:01.555183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:55431 (local address 127.17.31.62:38967)
0807 03:09:01.555724 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:01.555730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:01.556750 (+ 1020us) server_negotiation.cc:407] Connection header received
0807 03:09:01.558938 (+ 2188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:01.558944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:01.559029 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:01.559152 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:01.560589 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:01.561775 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:01.563429 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:01.564215 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:01.564359 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:01.568241 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:01.568247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:01.568250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:01.568283 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:01.571267 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:01.572180 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:01.572184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:01.572186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:01.572314 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:01.572788 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:01.572795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:01.572803 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:01.573086 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:01.573253 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:01.574073 (+ 820us) server_negotiation.cc:299] Negotiation successful
0807 03:09:01.574298 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":260,"threads_started":1}
I20260807 03:09:01.575893 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:01.555661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:38967 (local address 127.17.31.2:55431)
0807 03:09:01.556563 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:01.556610 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:01.558665 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:01.559346 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:01.559353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:01.559869 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:01.560410 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:01.560426 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:01.561928 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:01.561931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:01.563279 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:01.563291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:01.564513 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:01.564521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:01.564744 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:01.565515 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:01.565538 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:01.567976 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:01.571415 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:01.571424 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:01.571442 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:01.571989 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:01.572449 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:01.572454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:01.572458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:01.572658 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:01.573225 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:01.573233 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:01.573640 (+ 407us) client_negotiation.cc:769] Sending connection context
0807 03:09:01.573997 (+ 357us) client_negotiation.cc:240] Negotiation successful
0807 03:09:01.574565 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":405,"threads_started":1}
I20260807 03:09:01.577510 21582 heartbeater.cc:347] Connected to a master server at 127.17.31.62:38967
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
I20260807 03:09:01.577863 21582 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:01.578653 21582 heartbeater.cc:511] Master 127.17.31.62:38967 requested a full tablet report, sending...
I20260807 03:09:01.580202 21251 ts_manager.cc:194] Registered new tserver with Master: 6605bfc49def4bf690b666bf0799a615 (127.17.31.2:36969)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
I20260807 03:09:01.581228 21251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:55431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:09:01.605373 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:38967
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:46191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:01.926137 21589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:01.926530 21589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:01.926635 21589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:01.933805 21589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:01.933909 21589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:01.934026 21589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:09:01.944005 21589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:46191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:38967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:01.945386 21589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:01.946790 21589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:01.960704 21594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:01.960738 21597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:01.960708 21595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:01.961901 21589 server_base.cc:1034] running on GCE node
I20260807 03:09:01.962706 21589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:01.963724 21589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:01.964970 21589 hybrid_clock.cc:648] HybridClock initialized: now 1786072141964887 us; error 68 us; skew 500 ppm
I20260807 03:09:01.965327 21589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:01 dist-test-slave-tm1g krb5kdc[21205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072141, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:01.968647 21589 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:09:01.970395 21589 webserver.cc:460] Webserver started at http://127.17.31.3:44725/ using document root <none> and password file <none>
I20260807 03:09:01.971186 21589 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:01.971290 21589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:01.971568 21589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:01.975730 21589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "9b2b734158724ab7bee35d6786131337"
format_stamp: "Formatted at 2026-08-07 03:09:01 on dist-test-slave-tm1g"
server_key: "6987b0ca7ca13ad19bbccbdb50e5f582"
server_key_iv: "0c3a04be3f850149ef7ef5b411f0faad"
server_key_version: "encryptionkey@0"
I20260807 03:09:01.976572 21589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "9b2b734158724ab7bee35d6786131337"
format_stamp: "Formatted at 2026-08-07 03:09:01 on dist-test-slave-tm1g"
server_key: "6987b0ca7ca13ad19bbccbdb50e5f582"
server_key_iv: "0c3a04be3f850149ef7ef5b411f0faad"
server_key_version: "encryptionkey@0"
I20260807 03:09:01.981863 21589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 03:09:01.985450 21604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:01.987077 21589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:09:01.987210 21589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "9b2b734158724ab7bee35d6786131337"
format_stamp: "Formatted at 2026-08-07 03:09:01 on dist-test-slave-tm1g"
server_key: "6987b0ca7ca13ad19bbccbdb50e5f582"
server_key_iv: "0c3a04be3f850149ef7ef5b411f0faad"
server_key_version: "encryptionkey@0"
I20260807 03:09:01.987388 21589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:02.023684 21589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:02.030807 21589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:02.031147 21589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:02.032405 21589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:02.034206 21589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:02.034287 21589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:02.034482 21589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:02.034550 21589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:02.059598 21444 heartbeater.cc:503] Master 127.17.31.62:38967 was elected leader, sending a full tablet report...
I20260807 03:09:02.069686 21589 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:45897
I20260807 03:09:02.069721 21721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:45897 every 8 connection(s)
I20260807 03:09:02.071210 21589 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:09:02.075935 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 21589
I20260807 03:09:02.076129 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:09:02.076597 17532 external_mini_cluster.cc:1428] Setting key 43ad9ae0568b10fbb196e1f17acfdfa8
Aug 07 03:09:02 dist-test-slave-tm1g krb5kdc[21205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072141, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:02.091820 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:02.073857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:36873 (local address 127.17.31.62:38967)
0807 03:09:02.074051 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:02.074056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:02.075351 (+ 1295us) server_negotiation.cc:407] Connection header received
0807 03:09:02.077462 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:02.077468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:02.077591 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:02.077723 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:02.079350 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:02.080100 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:02.081685 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:02.082091 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:02.082234 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:02.085714 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:02.085725 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:02.085728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:02.085762 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:02.088459 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:02.089210 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:02.089215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:02.089219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:02.089353 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:02.089803 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:02.089814 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:02.089817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:02.090112 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:02.090284 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:02.091169 (+ 885us) server_negotiation.cc:299] Negotiation successful
0807 03:09:02.091411 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 03:09:02.092952 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:02.074473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:38967 (local address 127.17.31.3:36873)
0807 03:09:02.075178 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:02.075223 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:02.077159 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:02.077964 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:02.077973 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:02.078597 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:02.079168 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:02.079187 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:02.080265 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:02.080270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:02.081530 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:02.081541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:02.082377 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:02.082386 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:02.082623 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:02.083258 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:02.083284 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:02.085488 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:02.088608 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:02.088617 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:02.088634 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:02.089072 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:02.089472 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:02.089478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:02.089482 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:02.089676 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:02.090253 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:02.090261 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:02.090772 (+ 511us) client_negotiation.cc:769] Sending connection context
0807 03:09:02.091056 (+ 284us) client_negotiation.cc:240] Negotiation successful
0807 03:09:02.091622 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":287,"threads_started":1}
I20260807 03:09:02.094265 21722 heartbeater.cc:347] Connected to a master server at 127.17.31.62:38967
I20260807 03:09:02.094641 21722 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:02.095392 21722 heartbeater.cc:511] Master 127.17.31.62:38967 requested a full tablet report, sending...
I20260807 03:09:02.096854 21251 ts_manager.cc:194] Registered new tserver with Master: 9b2b734158724ab7bee35d6786131337 (127.17.31.3:45897)
I20260807 03:09:02.097687 21251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:36873
I20260807 03:09:02.104821 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 03:09:02 dist-test-slave-tm1g krb5kdc[21205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 03:09:02 dist-test-slave-tm1g krb5kdc[21205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:09:02.157569 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:02.142136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.17.31.1:39953)
0807 03:09:02.142937 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:02.142948 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:02.142981 (+ 33us) server_negotiation.cc:407] Connection header received
0807 03:09:02.143084 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:02.143095 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:02.143480 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:02.143701 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:02.145116 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:02.146180 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:02.147481 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:02.148049 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:02.148159 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:02.152113 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:02.152124 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:02.152140 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:02.152173 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:02.154926 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:02.155584 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:02.155590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:02.155593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:02.155712 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:02.156034 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:02.156037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:02.156039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:02.156364 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:02.156590 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:02.156856 (+ 266us) server_negotiation.cc:299] Negotiation successful
0807 03:09:02.157091 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":1470,"threads_started":1}
Aug 07 03:09:02 dist-test-slave-tm1g krb5kdc[21205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:02.176517 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:02.162150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.17.31.62:38967)
0807 03:09:02.162420 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:02.162427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:02.162592 (+ 165us) server_negotiation.cc:407] Connection header received
0807 03:09:02.162794 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:02.162800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:02.162898 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:02.163017 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:02.164327 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:02.165058 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:02.166429 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:02.166869 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:02.167014 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:02.171079 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:02.171093 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:02.171097 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:02.171160 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:02.173630 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:02.174295 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:02.174322 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:02.174327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:02.174553 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:02.174986 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:02.174991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:02.174994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:02.175322 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:02.175603 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:02.175950 (+ 347us) server_negotiation.cc:299] Negotiation successful
0807 03:09:02.176189 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260807 03:09:02.177105 21251 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34184
I20260807 03:09:02.178156 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 21314
I20260807 03:09:02.190567 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:02.179171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.17.31.1:39953)
0807 03:09:02.179394 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:02.179402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:02.179509 (+ 107us) server_negotiation.cc:407] Connection header received
0807 03:09:02.179641 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:02.179644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:02.179733 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:02.179810 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:02.180968 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:02.181658 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:02.182937 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:02.183341 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:02.183451 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:02.185697 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:02.185702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:02.185705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:02.185735 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:02.188294 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:02.188885 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:02.188889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:02.188891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:02.189007 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:02.189331 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:02.189334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:02.189336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:02.189581 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:02.189755 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:02.189997 (+ 242us) server_negotiation.cc:299] Negotiation successful
0807 03:09:02.190159 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 03:09:02.191277 21439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:02.272346 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 21314
I20260807 03:09:02.290026 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 21450
Aug 07 03:09:02 dist-test-slave-tm1g krb5kdc[21205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:09:02.305668 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:02.291122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49408 (local address 127.17.31.2:36969)
0807 03:09:02.291655 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:02.291662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:02.291688 (+ 26us) server_negotiation.cc:407] Connection header received
0807 03:09:02.291748 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:02.291754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:02.291974 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:02.292162 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:02.293279 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:02.294251 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:02.295524 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:02.296050 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:02.296145 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:02.300315 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:02.300326 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:02.300339 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:02.300380 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:02.303198 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:02.303822 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:02.303830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:02.303832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:02.303949 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:02.304335 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:02.304339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:02.304340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:02.304661 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:02.304873 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:02.305157 (+ 284us) server_negotiation.cc:299] Negotiation successful
0807 03:09:02.305357 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":269,"threads_started":1}
I20260807 03:09:02.306985 21577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:02.390172 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 21450
I20260807 03:09:02.407711 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 21589
Aug 07 03:09:02 dist-test-slave-tm1g krb5kdc[21205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:09:02.423674 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:02.408777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44696 (local address 127.17.31.3:45897)
0807 03:09:02.409363 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:02.409370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:02.409399 (+ 29us) server_negotiation.cc:407] Connection header received
0807 03:09:02.409461 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:02.409467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:02.409726 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:02.409859 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:02.411366 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:02.412390 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:02.413609 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:02.414147 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:02.414241 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:02.418520 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:02.418530 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:02.418544 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:02.418575 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:02.421170 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:02.421789 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:02.421797 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:02.421799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:02.421909 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:02.422213 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:02.422217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:02.422219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:02.422608 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:02.422818 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:02.423119 (+ 301us) server_negotiation.cc:299] Negotiation successful
0807 03:09:02.423306 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":282,"threads_started":1}
I20260807 03:09:02.425004 21717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:02.510612 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 21589
I20260807 03:09:02.527613 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 21219
I20260807 03:09:02.528815 21281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:02.631626 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 21219
2026-08-07T03:09:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3940 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:09:02 dist-test-slave-tm1g krb5kdc[21741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:09:02 dist-test-slave-tm1g krb5kdc[21741](info): set up 2 sockets
Aug 07 03:09:02 dist-test-slave-tm1g krb5kdc[21741](info): commencing operation
krb5kdc: starting...
W20260807 03:09:03.736063 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:09:03 dist-test-slave-tm1g krb5kdc[21741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:09:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:09:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:09:03.937925 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:40631
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:40631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:04.254683 21756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:04.255012 21756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:04.255190 21756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:04.262156 21756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:09:04.262301 21756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:04.262434 21756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:04.262533 21756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:09:04.262629 21756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:09:04.272073 21756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:40631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:40631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:04.273337 21756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:04.274669 21756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:04.286490 21761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:04.286573 21762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:04.286546 21764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:04.287365 21756 server_base.cc:1034] running on GCE node
I20260807 03:09:04.288093 21756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:04.289366 21756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:04.290589 21756 hybrid_clock.cc:648] HybridClock initialized: now 1786072144290562 us; error 45 us; skew 500 ppm
I20260807 03:09:04.290912 21756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:04 dist-test-slave-tm1g krb5kdc[21741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072144, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:04.293915 21756 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:09:04.295133 21756 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:04.295260 21756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:04.295593 21756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:04.298446 21756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "b1a13bad651841f5beb06baf85a9124d"
format_stamp: "Formatted at 2026-08-07 03:09:04 on dist-test-slave-tm1g"
server_key: "03e3f0fb01e8831ff2b1d4d5324b20b2"
server_key_iv: "a8bf9a6ae32c85dff567d784810fcfdd"
server_key_version: "encryptionkey@0"
I20260807 03:09:04.299230 21756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "b1a13bad651841f5beb06baf85a9124d"
format_stamp: "Formatted at 2026-08-07 03:09:04 on dist-test-slave-tm1g"
server_key: "03e3f0fb01e8831ff2b1d4d5324b20b2"
server_key_iv: "a8bf9a6ae32c85dff567d784810fcfdd"
server_key_version: "encryptionkey@0"
I20260807 03:09:04.303926 21756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260807 03:09:04.307230 21771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:04.308478 21756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:09:04.308668 21756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "b1a13bad651841f5beb06baf85a9124d"
format_stamp: "Formatted at 2026-08-07 03:09:04 on dist-test-slave-tm1g"
server_key: "03e3f0fb01e8831ff2b1d4d5324b20b2"
server_key_iv: "a8bf9a6ae32c85dff567d784810fcfdd"
server_key_version: "encryptionkey@0"
I20260807 03:09:04.308888 21756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:04.332083 21756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:04.336028 21756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:04.336295 21756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:04.354429 21756 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:40631
I20260807 03:09:04.354511 21824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:40631 every 8 connection(s)
I20260807 03:09:04.355297 21756 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:09:04.358727 21825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:04.365667 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 21756
I20260807 03:09:04.365875 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:09:04.366590 17532 external_mini_cluster.cc:1428] Setting key 29c9dad12bc2a935d89bfeff18610a98
I20260807 03:09:04.368799 21825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d: Bootstrap starting.
I20260807 03:09:04.372164 21825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:04.373464 21825 log.cc:824] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:04.376626 21825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d: No bootstrap required, opened a new log
Aug 07 03:09:04 dist-test-slave-tm1g krb5kdc[21741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:04.385591 21825 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a13bad651841f5beb06baf85a9124d" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40631 } }
I20260807 03:09:04.385939 21825 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:04.386070 21825 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1a13bad651841f5beb06baf85a9124d, State: Initialized, Role: FOLLOWER
I20260807 03:09:04.386767 21825 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a13bad651841f5beb06baf85a9124d" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40631 } }
I20260807 03:09:04.387072 21825 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:09:04.387243 21825 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:09:04.387445 21825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:04.389685 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:04.368521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.17.31.62:40631)
0807 03:09:04.369251 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:04.369261 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:04.369299 (+ 38us) server_negotiation.cc:407] Connection header received
0807 03:09:04.371120 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:04.371141 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:04.371502 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:04.371786 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:04.373780 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:04.375242 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:04.377035 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:04.377664 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:04.377782 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:04.382384 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:04.382392 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:04.382410 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:04.382442 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:04.385404 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:04.386167 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:04.386175 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:04.386190 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:04.386455 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:04.386931 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:04.386934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:04.386936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:04.387626 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:04.387863 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:04.388325 (+ 462us) server_negotiation.cc:299] Negotiation successful
0807 03:09:04.388670 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":546,"thread_start_us":344,"threads_started":1}
I20260807 03:09:04.390203 21825 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a13bad651841f5beb06baf85a9124d" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40631 } }
I20260807 03:09:04.390758 21825 leader_election.cc:302] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1a13bad651841f5beb06baf85a9124d; no voters:
I20260807 03:09:04.391371 21825 leader_election.cc:288] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:09:04.391444 21831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:04.393373 21825 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:09:04.395535 21831 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [term 1 LEADER]: Becoming Leader. State: Replica: b1a13bad651841f5beb06baf85a9124d, State: Running, Role: LEADER
I20260807 03:09:04.396070 21831 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a13bad651841f5beb06baf85a9124d" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40631 } }
I20260807 03:09:04.399192 21832 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1a13bad651841f5beb06baf85a9124d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a13bad651841f5beb06baf85a9124d" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40631 } } }
I20260807 03:09:04.399953 21832 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:04.399241 21833 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1a13bad651841f5beb06baf85a9124d. Latest consensus state: current_term: 1 leader_uuid: "b1a13bad651841f5beb06baf85a9124d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1a13bad651841f5beb06baf85a9124d" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40631 } } }
I20260807 03:09:04.400820 21833 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:04.401629 21841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:09:04.405877 21841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:09:04.413786 21841 catalog_manager.cc:1347] Generated new cluster ID: 76e74cba1f52494c8146d71ad45da27c
I20260807 03:09:04.413947 21841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:09:04.430719 21841 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:09:04.431972 21841 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:09:04.437911 21841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d: Generated new TSK 0
I20260807 03:09:04.438694 21841 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:09:04.494134 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:40631
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:04.814719 21855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:04.814970 21855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:04.815140 21855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:04.822242 21855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:04.822355 21855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:04.822543 21855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:09:04.832258 21855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:40631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:04.833552 21855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:04.834909 21855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:04.847911 21861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:04.847852 21860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:04.848105 21863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:04.848903 21855 server_base.cc:1034] running on GCE node
I20260807 03:09:04.849426 21855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:04.850266 21855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:04.851540 21855 hybrid_clock.cc:648] HybridClock initialized: now 1786072144851491 us; error 46 us; skew 500 ppm
I20260807 03:09:04.851847 21855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:04 dist-test-slave-tm1g krb5kdc[21741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072144, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:04.854872 21855 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:09:04.856040 21855 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:04.856175 21855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:04.856491 21855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:04.859361 21855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "1cbde5fbf0874c8c92df89b8350cb2f5"
format_stamp: "Formatted at 2026-08-07 03:09:04 on dist-test-slave-tm1g"
server_key: "fe4b10abdf7a177eef09407ca49e074a"
server_key_iv: "aa749b1a0ae20281a185c615363e1fb5"
server_key_version: "encryptionkey@0"
I20260807 03:09:04.860179 21855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "1cbde5fbf0874c8c92df89b8350cb2f5"
format_stamp: "Formatted at 2026-08-07 03:09:04 on dist-test-slave-tm1g"
server_key: "fe4b10abdf7a177eef09407ca49e074a"
server_key_iv: "aa749b1a0ae20281a185c615363e1fb5"
server_key_version: "encryptionkey@0"
I20260807 03:09:04.864950 21855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 03:09:04.868240 21870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:04.869475 21855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 03:09:04.869649 21855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "1cbde5fbf0874c8c92df89b8350cb2f5"
format_stamp: "Formatted at 2026-08-07 03:09:04 on dist-test-slave-tm1g"
server_key: "fe4b10abdf7a177eef09407ca49e074a"
server_key_iv: "aa749b1a0ae20281a185c615363e1fb5"
server_key_version: "encryptionkey@0"
I20260807 03:09:04.869859 21855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:04.898165 21855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:04.902841 21855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:04.903091 21855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:04.904016 21855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:04.905278 21855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:04.905402 21855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:04.905480 21855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:04.905535 21855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:04.939337 21855 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:40535
I20260807 03:09:04.939422 21994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:40535 every 8 connection(s)
I20260807 03:09:04.940230 21855 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:09:04.943493 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 21855
I20260807 03:09:04.943734 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:09:04.944581 17532 external_mini_cluster.cc:1428] Setting key d4613a81f5503d54c5236a568eb42d60
Aug 07 03:09:04 dist-test-slave-tm1g krb5kdc[21741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072144, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:04.963528 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:04.942730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:58115 (local address 127.17.31.62:40631)
0807 03:09:04.943164 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:04.943169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:04.944295 (+ 1126us) server_negotiation.cc:407] Connection header received
0807 03:09:04.946387 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:04.946393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:04.946476 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:04.946600 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:04.948626 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:04.949316 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:04.951850 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:04.952239 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:04.952333 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:04.956479 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:04.956488 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:04.956496 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:04.956547 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:04.960140 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:04.960931 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:04.960939 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:04.960945 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:04.961092 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:04.961497 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:04.961504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:04.961510 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:04.961892 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:04.962042 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:04.962793 (+ 751us) server_negotiation.cc:299] Negotiation successful
0807 03:09:04.963059 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":244,"threads_started":1}
I20260807 03:09:04.964087 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:04.943286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40631 (local address 127.17.31.1:58115)
0807 03:09:04.944134 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:04.944178 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:04.946046 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:04.946876 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:04.946887 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:04.947637 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:04.948446 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:04.948473 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:04.949487 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:04.949495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:04.951689 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:04.951706 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:04.952470 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:04.952479 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:04.952807 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:04.953549 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:04.953589 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:04.956255 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:04.960377 (+ 4122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:04.960383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:04.960395 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:04.960792 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:04.961207 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:04.961210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:04.961212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:04.961371 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:04.962020 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:04.962025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:04.962375 (+ 350us) client_negotiation.cc:769] Sending connection context
0807 03:09:04.962628 (+ 253us) client_negotiation.cc:240] Negotiation successful
0807 03:09:04.962994 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":364,"threads_started":1}
I20260807 03:09:04.965732 21995 heartbeater.cc:347] Connected to a master server at 127.17.31.62:40631
I20260807 03:09:04.966099 21995 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:04.966955 21995 heartbeater.cc:511] Master 127.17.31.62:40631 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
I20260807 03:09:04.969174 21788 ts_manager.cc:194] Registered new tserver with Master: 1cbde5fbf0874c8c92df89b8350cb2f5 (127.17.31.1:40535)
I20260807 03:09:04.970750 21788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:58115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:09:04.997002 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:40631
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:05.321774 22001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:05.322115 22001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:05.322297 22001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:05.329691 22001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:05.329860 22001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:05.330060 22001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:09:05.339911 22001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:40631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:05.341302 22001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:05.342727 22001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:05.355770 22006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:05.355747 22007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:05.358629 22009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:05.360265 22001 server_base.cc:1034] running on GCE node
I20260807 03:09:05.360859 22001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:05.361878 22001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:05.363101 22001 hybrid_clock.cc:648] HybridClock initialized: now 1786072145363049 us; error 45 us; skew 500 ppm
I20260807 03:09:05.363431 22001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:05 dist-test-slave-tm1g krb5kdc[21741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072145, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:05.366783 22001 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:09:05.368199 22001 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:05.368289 22001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:05.368604 22001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:05.371609 22001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "008431a1853e4bcbb507331be0442284"
format_stamp: "Formatted at 2026-08-07 03:09:05 on dist-test-slave-tm1g"
server_key: "eb9d403b2f69a5dcb578a1e55ac086a0"
server_key_iv: "ab9ef4fb641017bfcb7e9cb7cbf88314"
server_key_version: "encryptionkey@0"
I20260807 03:09:05.372428 22001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "008431a1853e4bcbb507331be0442284"
format_stamp: "Formatted at 2026-08-07 03:09:05 on dist-test-slave-tm1g"
server_key: "eb9d403b2f69a5dcb578a1e55ac086a0"
server_key_iv: "ab9ef4fb641017bfcb7e9cb7cbf88314"
server_key_version: "encryptionkey@0"
I20260807 03:09:05.377786 22001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:09:05.381299 22015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:05.382741 22001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:09:05.382931 22001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "008431a1853e4bcbb507331be0442284"
format_stamp: "Formatted at 2026-08-07 03:09:05 on dist-test-slave-tm1g"
server_key: "eb9d403b2f69a5dcb578a1e55ac086a0"
server_key_iv: "ab9ef4fb641017bfcb7e9cb7cbf88314"
server_key_version: "encryptionkey@0"
I20260807 03:09:05.383152 22001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:05.426023 22001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:05.430853 22001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:05.431200 22001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:05.432327 22001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:05.433634 22001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:05.433727 22001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:05.433808 22001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:05.433874 22001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:05.468237 22001 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:41401
I20260807 03:09:05.468322 22135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:41401 every 8 connection(s)
I20260807 03:09:05.469125 22001 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:09:05.477273 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 22001
I20260807 03:09:05.477447 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:09:05.477999 17532 external_mini_cluster.cc:1428] Setting key c1b76a1105438ff69f528bcf70eaac8a
Aug 07 03:09:05 dist-test-slave-tm1g krb5kdc[21741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072145, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:05.490727 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:05.471600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:50377 (local address 127.17.31.62:40631)
0807 03:09:05.472072 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:05.472077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:05.472874 (+ 797us) server_negotiation.cc:407] Connection header received
0807 03:09:05.474849 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:05.474855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:05.474948 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:05.475079 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:05.476563 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:05.477628 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:05.479504 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:05.480116 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:05.480250 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:05.483678 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:05.483687 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:05.483695 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:05.483739 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:05.487410 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:05.488213 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:05.488220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:05.488225 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:05.488389 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:05.488818 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:05.488826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:05.488829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:05.489213 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:05.489327 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:05.490212 (+ 885us) server_negotiation.cc:299] Negotiation successful
0807 03:09:05.490451 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":258,"threads_started":1}
I20260807 03:09:05.492240 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:05.472010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40631 (local address 127.17.31.2:50377)
0807 03:09:05.472711 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:05.472749 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:05.474560 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:05.475303 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:05.475311 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:05.475808 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:05.476355 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:05.476371 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:05.477788 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:05.477792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:05.479342 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:05.479362 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:05.480340 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:05.480346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:05.480538 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:05.481243 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:05.481265 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:05.483433 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:05.487541 (+ 4108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:05.487547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:05.487559 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:05.488056 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:05.488503 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:05.488507 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:05.488511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:05.488688 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:05.489349 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:05.489359 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:05.489806 (+ 447us) client_negotiation.cc:769] Sending connection context
0807 03:09:05.490125 (+ 319us) client_negotiation.cc:240] Negotiation successful
0807 03:09:05.490718 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":286,"threads_started":1}
I20260807 03:09:05.493856 22136 heartbeater.cc:347] Connected to a master server at 127.17.31.62:40631
I20260807 03:09:05.494233 22136 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:05.494967 22136 heartbeater.cc:511] Master 127.17.31.62:40631 requested a full tablet report, sending...
I20260807 03:09:05.496400 21788 ts_manager.cc:194] Registered new tserver with Master: 008431a1853e4bcbb507331be0442284 (127.17.31.2:41401)
I20260807 03:09:05.497200 21788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:50377
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:09:05.529560 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:40631
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:34999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:05.867908 22142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:05.868209 22142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:05.868383 22142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:05.875520 22142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:05.875659 22142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:05.875849 22142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:09:05.885074 22142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:34999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:40631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:05.886468 22142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:05.887936 22142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:05.900841 22147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:05.900985 22150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:05.901135 22148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:05.902021 22142 server_base.cc:1034] running on GCE node
I20260807 03:09:05.902665 22142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:05.903561 22142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:05.904789 22142 hybrid_clock.cc:648] HybridClock initialized: now 1786072145904747 us; error 59 us; skew 500 ppm
I20260807 03:09:05.905105 22142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:05 dist-test-slave-tm1g krb5kdc[21741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072145, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:05.908206 22142 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:09:05.909344 22142 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:05.909480 22142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:05.909788 22142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:05.912678 22142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "198c92cba767480cbbc15b817f47ae28"
format_stamp: "Formatted at 2026-08-07 03:09:05 on dist-test-slave-tm1g"
server_key: "bb0f4620aba694d95026532cd7f5b8e4"
server_key_iv: "7b5ded9e6b9bd28e650433eef98c5d6f"
server_key_version: "encryptionkey@0"
I20260807 03:09:05.913591 22142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "198c92cba767480cbbc15b817f47ae28"
format_stamp: "Formatted at 2026-08-07 03:09:05 on dist-test-slave-tm1g"
server_key: "bb0f4620aba694d95026532cd7f5b8e4"
server_key_iv: "7b5ded9e6b9bd28e650433eef98c5d6f"
server_key_version: "encryptionkey@0"
I20260807 03:09:05.918435 22142 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260807 03:09:05.921730 22156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:05.923228 22142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:09:05.923525 22142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "198c92cba767480cbbc15b817f47ae28"
format_stamp: "Formatted at 2026-08-07 03:09:05 on dist-test-slave-tm1g"
server_key: "bb0f4620aba694d95026532cd7f5b8e4"
server_key_iv: "7b5ded9e6b9bd28e650433eef98c5d6f"
server_key_version: "encryptionkey@0"
I20260807 03:09:05.923751 22142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:05.938093 22142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:05.942444 22142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:05.942709 22142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:05.943589 22142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:05.944849 22142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:05.944978 22142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:05.945129 22142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:05.945222 22142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:05.973703 21995 heartbeater.cc:503] Master 127.17.31.62:40631 was elected leader, sending a full tablet report...
I20260807 03:09:05.979784 22142 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:42595
I20260807 03:09:05.979866 22275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:42595 every 8 connection(s)
I20260807 03:09:05.980687 22142 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:09:05.988314 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 22142
I20260807 03:09:05.988492 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:09:05.989138 17532 external_mini_cluster.cc:1428] Setting key 91256c0a818cbef37a0c7906fddf92ce
Aug 07 03:09:05 dist-test-slave-tm1g krb5kdc[21741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072145, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:06.001724 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:05.983083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:45349 (local address 127.17.31.62:40631)
0807 03:09:05.983299 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:05.983305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:05.984429 (+ 1124us) server_negotiation.cc:407] Connection header received
0807 03:09:05.986116 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:05.986121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:05.986256 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:05.986450 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:05.988735 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:05.989918 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:05.992143 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:05.992526 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:05.992620 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:05.996100 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:05.996106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:05.996109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:05.996140 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:05.999079 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:05.999848 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:05.999852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:05.999855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:05.999973 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:06.000354 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:06.000357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:06.000359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:06.000636 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:06.000744 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:06.001322 (+ 578us) server_negotiation.cc:299] Negotiation successful
0807 03:09:06.001480 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 03:09:06.002866 22278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:05.983574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40631 (local address 127.17.31.3:45349)
0807 03:09:05.984251 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:05.984294 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:05.985882 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:05.986733 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:05.986747 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:05.987621 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:05.988490 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:05.988518 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:05.990111 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:05.990117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:05.991999 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:05.992017 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:05.992720 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:05.992724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:05.992901 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:05.993617 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:05.993643 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:05.995914 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:05.999198 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:05.999205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:05.999215 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:05.999699 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:06.000075 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:06.000078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:06.000081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:06.000257 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:06.000741 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:06.000746 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:06.001017 (+ 271us) client_negotiation.cc:769] Sending connection context
0807 03:09:06.001234 (+ 217us) client_negotiation.cc:240] Negotiation successful
0807 03:09:06.001577 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":285,"threads_started":1}
I20260807 03:09:06.004431 22276 heartbeater.cc:347] Connected to a master server at 127.17.31.62:40631
I20260807 03:09:06.004719 22276 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:06.005376 22276 heartbeater.cc:511] Master 127.17.31.62:40631 requested a full tablet report, sending...
I20260807 03:09:06.006788 21788 ts_manager.cc:194] Registered new tserver with Master: 198c92cba767480cbbc15b817f47ae28 (127.17.31.3:42595)
I20260807 03:09:06.007570 21788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:45349
I20260807 03:09:06.016451 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:09:06.033589 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.021110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.17.31.62:40631)
0807 03:09:06.021348 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:06.021354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:06.021667 (+ 313us) server_negotiation.cc:407] Connection header received
0807 03:09:06.021770 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:06.021774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:06.021829 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:06.021904 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:06.023476 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.024194 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.025421 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.025810 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.025898 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:06.028217 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:06.028221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:06.028224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:06.028251 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:06.031024 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:06.031577 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:06.031581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:06.031583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:06.031702 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:06.032167 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:06.032170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:06.032172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:06.032445 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:06.032701 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:06.033056 (+ 355us) server_negotiation.cc:299] Negotiation successful
0807 03:09:06.033265 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 03:09:06.037592 21788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:09:06.044297 21788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:09:06.070255 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.055955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:40535 (local address 127.0.0.1:46298)
0807 03:09:06.058244 (+ 2289us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:06.058321 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:06.058526 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:06.062831 (+ 4305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:06.062841 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:06.062897 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:06.063548 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:06.063559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.065055 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.065061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:06.067396 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:06.067410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.069221 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.069228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:06.069412 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:06.069461 (+ 49us) client_negotiation.cc:769] Sending connection context
0807 03:09:06.069599 (+ 138us) client_negotiation.cc:240] Negotiation successful
0807 03:09:06.069789 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1965,"mutex_wait_us":138,"spinlock_wait_cycles":12544,"thread_start_us":1806,"threads_started":1}
I20260807 03:09:06.070575 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.059330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.17.31.2:41401)
0807 03:09:06.061069 (+ 1739us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:06.061081 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:06.062257 (+ 1176us) server_negotiation.cc:407] Connection header received
0807 03:09:06.062378 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:06.062386 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:06.062591 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:06.062779 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:06.064212 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.065746 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.068403 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.069436 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.069590 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:06.069779 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:06.070022 (+ 243us) server_negotiation.cc:299] Negotiation successful
0807 03:09:06.070192 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1526,"thread_start_us":1393,"threads_started":1}
I20260807 03:09:06.070878 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.056304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.17.31.1:40535)
0807 03:09:06.062297 (+ 5993us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:06.062327 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:06.062349 (+ 22us) server_negotiation.cc:407] Connection header received
0807 03:09:06.062427 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:06.062433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:06.062614 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:06.063292 (+ 678us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:06.063737 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.064909 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.067582 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.069029 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.069152 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:06.069382 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:06.070390 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0807 03:09:06.070551 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5813,"thread_start_us":5642,"threads_started":1}
I20260807 03:09:06.071192 22289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.056245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:42595 (local address 127.0.0.1:42094)
0807 03:09:06.061573 (+ 5328us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:06.061605 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:06.061878 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:06.062833 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:06.062841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:06.062891 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:06.063811 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:06.063822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.065822 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.065827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:06.068308 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:06.068322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.070385 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.070391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:06.070565 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:06.070595 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 03:09:06.070687 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 03:09:06.070803 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5155,"mutex_wait_us":219,"spinlock_wait_cycles":11264,"thread_start_us":4998,"threads_started":1}
I20260807 03:09:06.071723 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.056944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42094 (local address 127.17.31.3:42595)
0807 03:09:06.060532 (+ 3588us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:06.060545 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:06.061738 (+ 1193us) server_negotiation.cc:407] Connection header received
0807 03:09:06.062100 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:06.062108 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:06.062376 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:06.062513 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:06.064015 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.065664 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.068481 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.069876 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.069989 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:06.070189 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:06.071179 (+ 990us) server_negotiation.cc:299] Negotiation successful
0807 03:09:06.071378 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3355,"thread_start_us":2589,"threads_started":1}
I20260807 03:09:06.072269 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.058738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:41401 (local address 127.0.0.1:37488)
0807 03:09:06.061972 (+ 3234us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:06.061992 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:06.062128 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:06.063020 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:06.063029 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:06.063072 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:06.064041 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:06.064052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.065918 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.065922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:06.068250 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:06.068263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.069650 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.069656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:06.069819 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:06.069852 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 03:09:06.069958 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 03:09:06.070090 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3100,"mutex_wait_us":91,"spinlock_wait_cycles":1408,"thread_start_us":2923,"threads_started":1}
I20260807 03:09:06.074635 21922 tablet_service.cc:1467] Processing CreateTablet for tablet b873f83857014b19999957f9217f8808 (DEFAULT_TABLE table=test-table [id=5852b35e0c014317aed85db13920e436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:06.075958 21922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b873f83857014b19999957f9217f8808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:06.075501 22206 tablet_service.cc:1467] Processing CreateTablet for tablet b873f83857014b19999957f9217f8808 (DEFAULT_TABLE table=test-table [id=5852b35e0c014317aed85db13920e436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:06.075796 22065 tablet_service.cc:1467] Processing CreateTablet for tablet b873f83857014b19999957f9217f8808 (DEFAULT_TABLE table=test-table [id=5852b35e0c014317aed85db13920e436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:06.076867 22206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b873f83857014b19999957f9217f8808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:06.077025 22065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b873f83857014b19999957f9217f8808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:06.092800 22296 tablet_bootstrap.cc:492] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28: Bootstrap starting.
I20260807 03:09:06.095188 22294 tablet_bootstrap.cc:492] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284: Bootstrap starting.
I20260807 03:09:06.098129 22294 tablet_bootstrap.cc:654] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:06.098995 22295 tablet_bootstrap.cc:492] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5: Bootstrap starting.
I20260807 03:09:06.099510 22294 log.cc:824] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:06.101763 22296 tablet_bootstrap.cc:654] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:06.102356 22295 tablet_bootstrap.cc:654] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:06.102766 22294 tablet_bootstrap.cc:492] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284: No bootstrap required, opened a new log
I20260807 03:09:06.102880 22296 log.cc:824] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:06.103044 22294 ts_tablet_manager.cc:1397] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260807 03:09:06.103576 22295 log.cc:824] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:06.105978 22296 tablet_bootstrap.cc:492] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28: No bootstrap required, opened a new log
I20260807 03:09:06.106261 22296 ts_tablet_manager.cc:1397] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260807 03:09:06.112882 22295 tablet_bootstrap.cc:492] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5: No bootstrap required, opened a new log
I20260807 03:09:06.113144 22295 ts_tablet_manager.cc:1397] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260807 03:09:06.113970 22296 raft_consensus.cc:348] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbde5fbf0874c8c92df89b8350cb2f5" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 40535 } } peers { permanent_uuid: "198c92cba767480cbbc15b817f47ae28" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42595 } } peers { permanent_uuid: "008431a1853e4bcbb507331be0442284" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41401 } }
I20260807 03:09:06.114341 22296 raft_consensus.cc:374] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:06.114454 22296 raft_consensus.cc:729] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198c92cba767480cbbc15b817f47ae28, State: Initialized, Role: FOLLOWER
I20260807 03:09:06.115051 22296 consensus_queue.cc:260] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbde5fbf0874c8c92df89b8350cb2f5" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 40535 } } peers { permanent_uuid: "198c92cba767480cbbc15b817f47ae28" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42595 } } peers { permanent_uuid: "008431a1853e4bcbb507331be0442284" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41401 } }
I20260807 03:09:06.115031 22294 raft_consensus.cc:348] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbde5fbf0874c8c92df89b8350cb2f5" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 40535 } } peers { permanent_uuid: "198c92cba767480cbbc15b817f47ae28" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42595 } } peers { permanent_uuid: "008431a1853e4bcbb507331be0442284" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41401 } }
I20260807 03:09:06.115444 22294 raft_consensus.cc:374] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:06.115603 22294 raft_consensus.cc:729] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 008431a1853e4bcbb507331be0442284, State: Initialized, Role: FOLLOWER
I20260807 03:09:06.116185 22276 heartbeater.cc:503] Master 127.17.31.62:40631 was elected leader, sending a full tablet report...
I20260807 03:09:06.116595 22296 ts_tablet_manager.cc:1428] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260807 03:09:06.117111 22294 consensus_queue.cc:260] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbde5fbf0874c8c92df89b8350cb2f5" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 40535 } } peers { permanent_uuid: "198c92cba767480cbbc15b817f47ae28" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42595 } } peers { permanent_uuid: "008431a1853e4bcbb507331be0442284" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41401 } }
I20260807 03:09:06.118208 22136 heartbeater.cc:503] Master 127.17.31.62:40631 was elected leader, sending a full tablet report...
I20260807 03:09:06.121146 22294 ts_tablet_manager.cc:1428] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260807 03:09:06.123579 22295 raft_consensus.cc:348] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbde5fbf0874c8c92df89b8350cb2f5" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 40535 } } peers { permanent_uuid: "198c92cba767480cbbc15b817f47ae28" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42595 } } peers { permanent_uuid: "008431a1853e4bcbb507331be0442284" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41401 } }
I20260807 03:09:06.123901 22295 raft_consensus.cc:374] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:06.123996 22295 raft_consensus.cc:729] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbde5fbf0874c8c92df89b8350cb2f5, State: Initialized, Role: FOLLOWER
I20260807 03:09:06.124476 22295 consensus_queue.cc:260] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbde5fbf0874c8c92df89b8350cb2f5" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 40535 } } peers { permanent_uuid: "198c92cba767480cbbc15b817f47ae28" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42595 } } peers { permanent_uuid: "008431a1853e4bcbb507331be0442284" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41401 } }
I20260807 03:09:06.125737 22295 ts_tablet_manager.cc:1428] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5: Time spent starting tablet: real 0.012s user 0.009s sys 0.005s
I20260807 03:09:06.183106 22302 raft_consensus.cc:482] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:09:06.183419 22302 raft_consensus.cc:504] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbde5fbf0874c8c92df89b8350cb2f5" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 40535 } } peers { permanent_uuid: "198c92cba767480cbbc15b817f47ae28" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42595 } } peers { permanent_uuid: "008431a1853e4bcbb507331be0442284" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41401 } }
I20260807 03:09:06.185153 22302 leader_election.cc:288] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 198c92cba767480cbbc15b817f47ae28 (127.17.31.3:42595), 008431a1853e4bcbb507331be0442284 (127.17.31.2:41401)
I20260807 03:09:06.191887 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.186042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:57601 (local address 127.17.31.2:41401)
0807 03:09:06.186282 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:06.186289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:06.186602 (+ 313us) server_negotiation.cc:407] Connection header received
0807 03:09:06.186783 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:06.186788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:06.186849 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:06.186947 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:06.187578 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.188279 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.190056 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.190940 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.191106 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:06.191207 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:06.191502 (+ 295us) server_negotiation.cc:299] Negotiation successful
0807 03:09:06.191628 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260807 03:09:06.191946 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.185556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:53935 (local address 127.17.31.3:42595)
0807 03:09:06.185785 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:06.185794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:06.186293 (+ 499us) server_negotiation.cc:407] Connection header received
0807 03:09:06.186565 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:06.186570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:06.186639 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:06.186750 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:06.187390 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.188138 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.190030 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.190956 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.191119 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:06.191238 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:06.191581 (+ 343us) server_negotiation.cc:299] Negotiation successful
0807 03:09:06.191700 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 03:09:06.192267 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.185553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:41401 (local address 127.17.31.1:57601)
0807 03:09:06.186480 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:06.186496 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:06.186635 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:06.186966 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:06.186970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:06.186994 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:06.187424 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:06.187433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.188425 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.188431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:06.189893 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:06.189905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.191186 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.191192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:06.191338 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:06.191362 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 03:09:06.191477 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 03:09:06.191598 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":17,"thread_start_us":755,"threads_started":1}
I20260807 03:09:06.192333 22303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.185408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:42595 (local address 127.17.31.1:53935)
0807 03:09:06.186164 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:06.186186 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:06.186396 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:06.186768 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:06.186773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:06.186798 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:06.187225 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:06.187257 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.188281 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.188287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:06.189855 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:06.189870 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.191252 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.191261 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:06.191415 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:06.191437 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 03:09:06.191556 (+ 119us) client_negotiation.cc:240] Negotiation successful
0807 03:09:06.191680 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":33,"thread_start_us":375,"threads_started":1}
W20260807 03:09:06.192600 21996 tablet.cc:2369] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:09:06.193476 22088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b873f83857014b19999957f9217f8808" candidate_uuid: "1cbde5fbf0874c8c92df89b8350cb2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "008431a1853e4bcbb507331be0442284" is_pre_election: true
I20260807 03:09:06.193486 22227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b873f83857014b19999957f9217f8808" candidate_uuid: "1cbde5fbf0874c8c92df89b8350cb2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198c92cba767480cbbc15b817f47ae28" is_pre_election: true
I20260807 03:09:06.194034 22227 raft_consensus.cc:2454] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbde5fbf0874c8c92df89b8350cb2f5 in term 0.
I20260807 03:09:06.194036 22088 raft_consensus.cc:2454] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbde5fbf0874c8c92df89b8350cb2f5 in term 0.
I20260807 03:09:06.194713 21871 leader_election.cc:302] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 008431a1853e4bcbb507331be0442284, 1cbde5fbf0874c8c92df89b8350cb2f5; no voters:
I20260807 03:09:06.195158 22302 raft_consensus.cc:2781] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:09:06.195350 22302 raft_consensus.cc:482] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:09:06.195463 22302 raft_consensus.cc:3037] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:06.197583 22302 raft_consensus.cc:504] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbde5fbf0874c8c92df89b8350cb2f5" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 40535 } } peers { permanent_uuid: "198c92cba767480cbbc15b817f47ae28" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42595 } } peers { permanent_uuid: "008431a1853e4bcbb507331be0442284" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41401 } }
I20260807 03:09:06.198194 22302 leader_election.cc:288] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [CANDIDATE]: Term 1 election: Requested vote from peers 198c92cba767480cbbc15b817f47ae28 (127.17.31.3:42595), 008431a1853e4bcbb507331be0442284 (127.17.31.2:41401)
I20260807 03:09:06.198644 22227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b873f83857014b19999957f9217f8808" candidate_uuid: "1cbde5fbf0874c8c92df89b8350cb2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198c92cba767480cbbc15b817f47ae28"
I20260807 03:09:06.198669 22088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b873f83857014b19999957f9217f8808" candidate_uuid: "1cbde5fbf0874c8c92df89b8350cb2f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "008431a1853e4bcbb507331be0442284"
I20260807 03:09:06.198830 22227 raft_consensus.cc:3037] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:06.198884 22088 raft_consensus.cc:3037] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:06.201216 22227 raft_consensus.cc:2454] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbde5fbf0874c8c92df89b8350cb2f5 in term 1.
I20260807 03:09:06.201229 22088 raft_consensus.cc:2454] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbde5fbf0874c8c92df89b8350cb2f5 in term 1.
I20260807 03:09:06.201670 21873 leader_election.cc:302] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 008431a1853e4bcbb507331be0442284, 198c92cba767480cbbc15b817f47ae28, 1cbde5fbf0874c8c92df89b8350cb2f5; no voters:
I20260807 03:09:06.202008 22302 raft_consensus.cc:2781] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:06.202430 22302 raft_consensus.cc:686] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbde5fbf0874c8c92df89b8350cb2f5, State: Running, Role: LEADER
I20260807 03:09:06.202957 22302 consensus_queue.cc:237] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbde5fbf0874c8c92df89b8350cb2f5" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 40535 } } peers { permanent_uuid: "198c92cba767480cbbc15b817f47ae28" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42595 } } peers { permanent_uuid: "008431a1853e4bcbb507331be0442284" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41401 } }
I20260807 03:09:06.207100 21787 catalog_manager.cc:5495] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbde5fbf0874c8c92df89b8350cb2f5 (127.17.31.1). New cstate: current_term: 1 leader_uuid: "1cbde5fbf0874c8c92df89b8350cb2f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbde5fbf0874c8c92df89b8350cb2f5" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 40535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "198c92cba767480cbbc15b817f47ae28" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "008431a1853e4bcbb507331be0442284" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41401 } health_report { overall_health: UNKNOWN } } }
W20260807 03:09:06.221055 22137 tablet.cc:2369] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:09:06.223644 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.218390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46310 (local address 127.17.31.1:40535)
0807 03:09:06.218608 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:06.218614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:06.218760 (+ 146us) server_negotiation.cc:407] Connection header received
0807 03:09:06.219014 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:06.219018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:06.219096 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:06.219268 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:09:06.219662 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.220379 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.221768 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.222145 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.222235 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:06.222739 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:09:06.223191 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:09:06.223299 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:06.223401 (+ 102us) server_negotiation.cc:299] Negotiation successful
0807 03:09:06.223460 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 03:09:06.231088 22088 raft_consensus.cc:1261] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284 [term 1 FOLLOWER]: Refusing update from remote peer 1cbde5fbf0874c8c92df89b8350cb2f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:09:06.231163 22227 raft_consensus.cc:1261] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28 [term 1 FOLLOWER]: Refusing update from remote peer 1cbde5fbf0874c8c92df89b8350cb2f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:09:06.232038 22305 consensus_queue.cc:1035] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "008431a1853e4bcbb507331be0442284" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 03:09:06.232179 22277 tablet.cc:2369] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:09:06.232368 22302 consensus_queue.cc:1035] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "198c92cba767480cbbc15b817f47ae28" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:06.257571 21787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:09:06.257876 21787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:09:06.261402 21787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1a13bad651841f5beb06baf85a9124d: Sending DeleteTablet for 3 replicas of tablet b873f83857014b19999957f9217f8808
I20260807 03:09:06.262449 21922 tablet_service.cc:1514] Processing DeleteTablet for tablet b873f83857014b19999957f9217f8808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:06 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:46298
I20260807 03:09:06.262612 22206 tablet_service.cc:1514] Processing DeleteTablet for tablet b873f83857014b19999957f9217f8808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:06 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:42094
I20260807 03:09:06.263644 22065 tablet_service.cc:1514] Processing DeleteTablet for tablet b873f83857014b19999957f9217f8808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:06 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:37488
I20260807 03:09:06.263968 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 21855
Aug 07 03:09:06 dist-test-slave-tm1g krb5kdc[21741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:09:06.287889 22321 tablet_replica.cc:330] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284: stopping tablet replica
I20260807 03:09:06.288555 22321 raft_consensus.cc:2229] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:06.289321 22321 raft_consensus.cc:2258] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:06.292028 22321 ts_tablet_manager.cc:1905] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:06.295320 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.265287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.17.31.1:40535)
0807 03:09:06.265491 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:06.265497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:06.275632 (+ 10135us) server_negotiation.cc:407] Connection header received
0807 03:09:06.276065 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:06.276071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:06.276147 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:06.276226 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:06.278118 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.279378 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.281260 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.281988 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.282093 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:06.287596 (+ 5503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:06.287610 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:06.287628 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:06.287673 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:06.291577 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:06.292449 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:06.292458 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:06.292461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:06.292642 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:06.293129 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:06.293135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:06.293138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:06.293646 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:06.293925 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:06.294583 (+ 658us) server_negotiation.cc:299] Negotiation successful
0807 03:09:06.294886 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 03:09:06.296667 22319 tablet_replica.cc:330] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5: stopping tablet replica
I20260807 03:09:06.297060 21989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:06.298166 22320 tablet_replica.cc:330] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28: stopping tablet replica
I20260807 03:09:06.298851 22320 raft_consensus.cc:2229] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:06.299580 22320 raft_consensus.cc:2258] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:06.301553 22321 ts_tablet_manager.cc:1918] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:09:06.301735 22321 log.cc:1196] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/b873f83857014b19999957f9217f8808
I20260807 03:09:06.301891 22320 ts_tablet_manager.cc:1905] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:06.302289 22321 ts_tablet_manager.cc:1939] T b873f83857014b19999957f9217f8808 P 008431a1853e4bcbb507331be0442284: Deleting consensus metadata
I20260807 03:09:06.303799 21772 catalog_manager.cc:4841] TS 008431a1853e4bcbb507331be0442284 (127.17.31.2:41401): tablet b873f83857014b19999957f9217f8808 (table test-table [id=5852b35e0c014317aed85db13920e436]) successfully deleted
I20260807 03:09:06.309130 22320 ts_tablet_manager.cc:1918] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:09:06.309305 22320 log.cc:1196] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/b873f83857014b19999957f9217f8808
I20260807 03:09:06.309842 22320 ts_tablet_manager.cc:1939] T b873f83857014b19999957f9217f8808 P 198c92cba767480cbbc15b817f47ae28: Deleting consensus metadata
I20260807 03:09:06.311676 21774 catalog_manager.cc:4841] TS 198c92cba767480cbbc15b817f47ae28 (127.17.31.3:42595): tablet b873f83857014b19999957f9217f8808 (table test-table [id=5852b35e0c014317aed85db13920e436]) successfully deleted
I20260807 03:09:06.297528 22319 raft_consensus.cc:2229] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:09:06.407008 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 21855
I20260807 03:09:06.407007 22319 raft_consensus.cc:2258] T b873f83857014b19999957f9217f8808 P 1cbde5fbf0874c8c92df89b8350cb2f5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260807 03:09:06.426975 21772 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.31.1:40535 (error 108)
I20260807 03:09:06.427173 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 22001
I20260807 03:09:06.429018 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.428532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:40535 (local address 127.0.0.1:46326)
0807 03:09:06.428724 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:06.428841 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:40535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260807 03:09:06.429250 21772 catalog_manager.cc:4584] TS 1cbde5fbf0874c8c92df89b8350cb2f5 (127.17.31.1:40535): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b873f83857014b19999957f9217f8808: Network error: Client connection negotiation failed: client connection to 127.17.31.1:40535: connect: Connection refused (error 111)
Aug 07 03:09:06 dist-test-slave-tm1g krb5kdc[21741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:09:06.441615 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.428045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.17.31.2:41401)
0807 03:09:06.428211 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:06.428219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:06.428240 (+ 21us) server_negotiation.cc:407] Connection header received
0807 03:09:06.428520 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:06.428524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:06.428666 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:06.428775 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:06.430132 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.430930 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.432103 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.432467 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.432563 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:06.436587 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:06.436599 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:06.436613 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:06.436644 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:06.439256 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:06.439829 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:06.439835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:06.439838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:06.439953 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:06.440264 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:06.440267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:06.440269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:06.440614 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:06.440839 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:06.441085 (+ 246us) server_negotiation.cc:299] Negotiation successful
0807 03:09:06.441239 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 03:09:06.442278 22131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:06.482999 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.482474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:40535 (local address 127.0.0.1:46330)
0807 03:09:06.482692 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:06.482788 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:40535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260807 03:09:06.484074 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.483720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:40535 (local address 127.0.0.1:46342)
0807 03:09:06.483873 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:06.483957 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:40535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 03:09:06.531847 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 22001
I20260807 03:09:06.544565 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.544116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:40535 (local address 127.0.0.1:46344)
0807 03:09:06.544324 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:06.544405 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:40535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 03:09:06.545575 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.545227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:40535 (local address 127.0.0.1:46348)
0807 03:09:06.545391 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:06.545465 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:40535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 03:09:06.550450 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 22142
Aug 07 03:09:06 dist-test-slave-tm1g krb5kdc[21741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:09:06.564713 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.551173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.17.31.3:42595)
0807 03:09:06.551360 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:06.551364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:06.551382 (+ 18us) server_negotiation.cc:407] Connection header received
0807 03:09:06.551525 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:06.551529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:06.551603 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:06.551685 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:06.552915 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.553641 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.554892 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:06.555263 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:06.555357 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:06.559619 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:06.559632 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:06.559649 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:06.559679 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:06.562194 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:06.562910 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:06.562917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:06.562919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:06.563053 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:06.563391 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:06.563394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:06.563396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:06.563733 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:06.563938 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:06.564204 (+ 266us) server_negotiation.cc:299] Negotiation successful
0807 03:09:06.564363 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 03:09:06.565549 22270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:06.626533 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.626003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:40535 (local address 127.0.0.1:46360)
0807 03:09:06.626208 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:06.626353 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:40535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 03:09:06.627650 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:06.627241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:40535 (local address 127.0.0.1:46368)
0807 03:09:06.627450 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:06.627541 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:40535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260807 03:09:06.655617 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 22142
I20260807 03:09:06.673698 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 21756
I20260807 03:09:06.674854 21820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:06.783499 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 21756
2026-08-07T03:09:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4152 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-07T03:09:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:09:06Z Disabled control of system clock
I20260807 03:09:06.854957 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:35107
--webserver_interface=127.17.31.62
--webserver_port=0
--builtin_ntp_servers=127.17.31.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:35107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 03:09:07.186677 22331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:07.186950 22331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:07.187129 22331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:07.194651 22331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:09:07.194797 22331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:07.194867 22331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:07.194972 22331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:09:07.195055 22331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:09:07.204116 22331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:35107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:35107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:07.205451 22331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:07.207278 22331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:07.218825 22338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:07.218948 22341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:07.218873 22339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:07.219940 22331 server_base.cc:1034] running on GCE node
I20260807 03:09:07.220729 22331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:07.222153 22331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:07.223385 22331 hybrid_clock.cc:648] HybridClock initialized: now 1786072147223337 us; error 62 us; skew 500 ppm
I20260807 03:09:07.223724 22331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:09:07.225940 22331 webserver.cc:460] Webserver started at http://127.17.31.62:33489/ using document root <none> and password file <none>
I20260807 03:09:07.226702 22331 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:07.226850 22331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:07.227167 22331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:07.230014 22331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "c053f9d52ae74b2f8352f47f2fb1f18a"
format_stamp: "Formatted at 2026-08-07 03:09:07 on dist-test-slave-tm1g"
server_key: "11afe4ece559cf5b18be5ddea8a5a444"
server_key_iv: "ff66ecc888f3d8f23178e3081e0296d0"
server_key_version: "encryptionkey@0"
I20260807 03:09:07.230815 22331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "c053f9d52ae74b2f8352f47f2fb1f18a"
format_stamp: "Formatted at 2026-08-07 03:09:07 on dist-test-slave-tm1g"
server_key: "11afe4ece559cf5b18be5ddea8a5a444"
server_key_iv: "ff66ecc888f3d8f23178e3081e0296d0"
server_key_version: "encryptionkey@0"
I20260807 03:09:07.235644 22331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260807 03:09:07.238876 22348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:07.240124 22331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:09:07.240299 22331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "c053f9d52ae74b2f8352f47f2fb1f18a"
format_stamp: "Formatted at 2026-08-07 03:09:07 on dist-test-slave-tm1g"
server_key: "11afe4ece559cf5b18be5ddea8a5a444"
server_key_iv: "ff66ecc888f3d8f23178e3081e0296d0"
server_key_version: "encryptionkey@0"
I20260807 03:09:07.240509 22331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:07.246507 22331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:07.247241 22331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:07.247556 22331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:07.265707 22331 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:35107
I20260807 03:09:07.265738 22402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:35107 every 8 connection(s)
I20260807 03:09:07.267107 22331 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:09:07.270627 22403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:07.272517 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 22331
I20260807 03:09:07.272759 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:09:07.273401 17532 external_mini_cluster.cc:1428] Setting key 3b85cec6cf73e571329477f4828f8e6e
I20260807 03:09:07.281418 22406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:07.275415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.17.31.62:35107)
0807 03:09:07.276380 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:07.276390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:07.276431 (+ 41us) server_negotiation.cc:407] Connection header received
0807 03:09:07.278005 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:07.278029 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:07.278491 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:07.278809 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:07.279918 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:07.279924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:07.279935 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:07.279952 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:07.280010 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:07.280024 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:07.280218 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:07.280308 (+ 90us) server_negotiation.cc:299] Negotiation successful
0807 03:09:07.280567 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":403,"threads_started":1}
I20260807 03:09:07.282264 22403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a: Bootstrap starting.
I20260807 03:09:07.285285 22403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:07.286546 22403 log.cc:824] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:07.288923 22403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a: No bootstrap required, opened a new log
I20260807 03:09:07.296089 22403 raft_consensus.cc:348] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c053f9d52ae74b2f8352f47f2fb1f18a" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35107 } }
I20260807 03:09:07.296380 22403 raft_consensus.cc:374] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:07.296468 22403 raft_consensus.cc:729] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c053f9d52ae74b2f8352f47f2fb1f18a, State: Initialized, Role: FOLLOWER
I20260807 03:09:07.296969 22403 consensus_queue.cc:260] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c053f9d52ae74b2f8352f47f2fb1f18a" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35107 } }
I20260807 03:09:07.297194 22403 raft_consensus.cc:388] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:09:07.297345 22403 raft_consensus.cc:482] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:09:07.297515 22403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:07.299551 22403 raft_consensus.cc:504] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c053f9d52ae74b2f8352f47f2fb1f18a" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35107 } }
I20260807 03:09:07.300019 22403 leader_election.cc:302] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c053f9d52ae74b2f8352f47f2fb1f18a; no voters:
I20260807 03:09:07.300578 22403 leader_election.cc:288] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:09:07.300734 22408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:07.301478 22408 raft_consensus.cc:686] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [term 1 LEADER]: Becoming Leader. State: Replica: c053f9d52ae74b2f8352f47f2fb1f18a, State: Running, Role: LEADER
I20260807 03:09:07.302033 22408 consensus_queue.cc:237] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c053f9d52ae74b2f8352f47f2fb1f18a" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35107 } }
I20260807 03:09:07.302585 22403 sys_catalog.cc:564] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:09:07.304642 22410 sys_catalog.cc:455] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c053f9d52ae74b2f8352f47f2fb1f18a. Latest consensus state: current_term: 1 leader_uuid: "c053f9d52ae74b2f8352f47f2fb1f18a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c053f9d52ae74b2f8352f47f2fb1f18a" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35107 } } }
I20260807 03:09:07.305096 22410 sys_catalog.cc:458] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:07.306833 22415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:09:07.307214 22409 sys_catalog.cc:455] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c053f9d52ae74b2f8352f47f2fb1f18a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c053f9d52ae74b2f8352f47f2fb1f18a" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35107 } } }
I20260807 03:09:07.307574 22409 sys_catalog.cc:458] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:07.312294 22415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:09:07.322413 22415 catalog_manager.cc:1347] Generated new cluster ID: 32859cd128af4dee9759c5e10fe8ddec
I20260807 03:09:07.322533 22415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:09:07.341729 22415 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:09:07.342139 22415 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:09:07.350754 22415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a: Generated new TSK 0
I20260807 03:09:07.351511 22415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:09:07.362514 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35107
--builtin_ntp_servers=127.17.31.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 03:09:07.686182 22427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:07.686478 22427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:07.686647 22427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:07.693524 22427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:07.693611 22427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:07.693792 22427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:09:07.702922 22427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:07.704262 22427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:07.706012 22427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:07.718786 22435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:07.718864 22432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:07.718838 22433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:07.719699 22427 server_base.cc:1034] running on GCE node
I20260807 03:09:07.720266 22427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:07.721221 22427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:07.722489 22427 hybrid_clock.cc:648] HybridClock initialized: now 1786072147722424 us; error 45 us; skew 500 ppm
I20260807 03:09:07.722815 22427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:09:07.725040 22427 webserver.cc:460] Webserver started at http://127.17.31.1:35317/ using document root <none> and password file <none>
I20260807 03:09:07.725726 22427 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:07.725841 22427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:07.726179 22427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:07.728907 22427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "ade6cf5477a54fa5a940f17587aa6942"
format_stamp: "Formatted at 2026-08-07 03:09:07 on dist-test-slave-tm1g"
server_key: "91a15e92ea1ce7db0538750419e68dbd"
server_key_iv: "fbf51adc5238ba96762acbc1b3839648"
server_key_version: "encryptionkey@0"
I20260807 03:09:07.729684 22427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "ade6cf5477a54fa5a940f17587aa6942"
format_stamp: "Formatted at 2026-08-07 03:09:07 on dist-test-slave-tm1g"
server_key: "91a15e92ea1ce7db0538750419e68dbd"
server_key_iv: "fbf51adc5238ba96762acbc1b3839648"
server_key_version: "encryptionkey@0"
I20260807 03:09:07.734588 22427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 03:09:07.737785 22441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:07.739010 22427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 03:09:07.739187 22427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "ade6cf5477a54fa5a940f17587aa6942"
format_stamp: "Formatted at 2026-08-07 03:09:07 on dist-test-slave-tm1g"
server_key: "91a15e92ea1ce7db0538750419e68dbd"
server_key_iv: "fbf51adc5238ba96762acbc1b3839648"
server_key_version: "encryptionkey@0"
I20260807 03:09:07.739408 22427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:07.746399 22427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:07.747335 22427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:07.747632 22427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:07.749078 22427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:07.751128 22427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:07.751235 22427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:07.751343 22427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:07.751427 22427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:07.783838 22427 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:41805
I20260807 03:09:07.783928 22559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:41805 every 8 connection(s)
I20260807 03:09:07.785140 22427 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:09:07.792701 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 22427
I20260807 03:09:07.792933 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:09:07.792979 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:07.787643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:59377 (local address 127.17.31.62:35107)
0807 03:09:07.788098 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:07.788102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:07.789075 (+ 973us) server_negotiation.cc:407] Connection header received
0807 03:09:07.790804 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:07.790810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:07.790879 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:07.791011 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:07.792213 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:07.792219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:07.792224 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:07.792242 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:07.792292 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:07.792304 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:07.792440 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:07.792624 (+ 184us) server_negotiation.cc:299] Negotiation successful
0807 03:09:07.792691 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":255,"threads_started":1}
I20260807 03:09:07.793596 17532 external_mini_cluster.cc:1428] Setting key bb8b74b8c036cdf12f125f2e33cca797
I20260807 03:09:07.793885 22563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:07.788035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35107 (local address 127.17.31.1:59377)
0807 03:09:07.788914 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:07.788950 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:07.790540 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:07.791247 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:07.791254 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:07.791339 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:07.791928 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:09:07.791931 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:07.791972 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:09:07.792049 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:07.792402 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:07.792411 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:07.792419 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 03:09:07.792704 (+ 285us) client_negotiation.cc:240] Negotiation successful
0807 03:09:07.792946 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":268,"threads_started":1}
I20260807 03:09:07.795353 22560 heartbeater.cc:347] Connected to a master server at 127.17.31.62:35107
I20260807 03:09:07.795660 22560 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:07.796196 22560 heartbeater.cc:511] Master 127.17.31.62:35107 requested a full tablet report, sending...
I20260807 03:09:07.796653 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35107
--builtin_ntp_servers=127.17.31.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 03:09:07.798141 22366 ts_manager.cc:194] Registered new tserver with Master: ade6cf5477a54fa5a940f17587aa6942 (127.17.31.1:41805)
W20260807 03:09:08.115833 22564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:08.116084 22564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:08.116170 22564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:08.123086 22564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:08.123175 22564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:08.123353 22564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:09:08.132107 22564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:08.133369 22564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:08.135234 22564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:08.148240 22572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:08.148334 22570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:08.148277 22574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:08.149080 22564 server_base.cc:1034] running on GCE node
I20260807 03:09:08.149592 22564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:08.150502 22564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:08.151736 22564 hybrid_clock.cc:648] HybridClock initialized: now 1786072148151690 us; error 35 us; skew 500 ppm
I20260807 03:09:08.152084 22564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:09:08.154409 22564 webserver.cc:460] Webserver started at http://127.17.31.2:36513/ using document root <none> and password file <none>
I20260807 03:09:08.155105 22564 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:08.155223 22564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:08.155557 22564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:08.158430 22564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "33ac7808a2104ebcbb8a46ec8186c487"
format_stamp: "Formatted at 2026-08-07 03:09:08 on dist-test-slave-tm1g"
server_key: "9cb24d704cb23f296da83bfa741ff6a6"
server_key_iv: "dfc38e16e6c87a2d1bd564143e918287"
server_key_version: "encryptionkey@0"
I20260807 03:09:08.159222 22564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "33ac7808a2104ebcbb8a46ec8186c487"
format_stamp: "Formatted at 2026-08-07 03:09:08 on dist-test-slave-tm1g"
server_key: "9cb24d704cb23f296da83bfa741ff6a6"
server_key_iv: "dfc38e16e6c87a2d1bd564143e918287"
server_key_version: "encryptionkey@0"
I20260807 03:09:08.164157 22564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260807 03:09:08.167562 22580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:08.168794 22564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 03:09:08.168969 22564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "33ac7808a2104ebcbb8a46ec8186c487"
format_stamp: "Formatted at 2026-08-07 03:09:08 on dist-test-slave-tm1g"
server_key: "9cb24d704cb23f296da83bfa741ff6a6"
server_key_iv: "dfc38e16e6c87a2d1bd564143e918287"
server_key_version: "encryptionkey@0"
I20260807 03:09:08.169175 22564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:08.174705 22564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:08.175452 22564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:08.175726 22564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:08.176663 22564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:08.177977 22564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:08.178094 22564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:08.178226 22564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:08.178404 22564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:08.211730 22564 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:42023
I20260807 03:09:08.211781 22696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:42023 every 8 connection(s)
I20260807 03:09:08.213140 22564 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:09:08.217340 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 22564
I20260807 03:09:08.217540 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:09:08.218114 17532 external_mini_cluster.cc:1428] Setting key b698675a66981503478211d05e35dc8c
I20260807 03:09:08.221333 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35107
--builtin_ntp_servers=127.17.31.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 03:09:08.221927 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.215736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:45469 (local address 127.17.31.62:35107)
0807 03:09:08.215904 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:08.215909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:08.217402 (+ 1493us) server_negotiation.cc:407] Connection header received
0807 03:09:08.219482 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:08.219487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:08.219582 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:08.219698 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:08.221130 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:08.221134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:08.221137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:08.221153 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:08.221186 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:08.221197 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:08.221292 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:08.221609 (+ 317us) server_negotiation.cc:299] Negotiation successful
0807 03:09:08.221677 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 03:09:08.223187 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.216225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35107 (local address 127.17.31.2:45469)
0807 03:09:08.217221 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:08.217267 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:08.219217 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:08.219913 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:08.219922 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:08.220010 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:08.220774 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:09:08.220777 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:08.220827 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:09:08.220947 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:08.221309 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:08.221322 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:08.221331 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 03:09:08.221637 (+ 306us) client_negotiation.cc:240] Negotiation successful
0807 03:09:08.221889 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":285,"threads_started":1}
I20260807 03:09:08.224638 22697 heartbeater.cc:347] Connected to a master server at 127.17.31.62:35107
I20260807 03:09:08.224944 22697 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:08.225659 22697 heartbeater.cc:511] Master 127.17.31.62:35107 requested a full tablet report, sending...
I20260807 03:09:08.227296 22366 ts_manager.cc:194] Registered new tserver with Master: 33ac7808a2104ebcbb8a46ec8186c487 (127.17.31.2:42023)
W20260807 03:09:08.553411 22700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:08.553674 22700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:08.553763 22700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:08.560847 22700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:08.560941 22700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:08.561048 22700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:09:08.569880 22700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:08.571185 22700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:08.572860 22700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:08.585626 22709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:08.585723 22707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:08.586114 22706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:08.587239 22700 server_base.cc:1034] running on GCE node
I20260807 03:09:08.587808 22700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:08.588747 22700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:08.589967 22700 hybrid_clock.cc:648] HybridClock initialized: now 1786072148589922 us; error 62 us; skew 500 ppm
I20260807 03:09:08.590265 22700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:09:08.592447 22700 webserver.cc:460] Webserver started at http://127.17.31.3:34259/ using document root <none> and password file <none>
I20260807 03:09:08.593204 22700 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:08.593334 22700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:08.593683 22700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:08.596472 22700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "b7d47daa49da43a3b5bd7d4c53113199"
format_stamp: "Formatted at 2026-08-07 03:09:08 on dist-test-slave-tm1g"
server_key: "690c5d7402a0ad26fc86500ceaa1172d"
server_key_iv: "c7114cab9ce57b0158d4728f4eb2dfc1"
server_key_version: "encryptionkey@0"
I20260807 03:09:08.597261 22700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "b7d47daa49da43a3b5bd7d4c53113199"
format_stamp: "Formatted at 2026-08-07 03:09:08 on dist-test-slave-tm1g"
server_key: "690c5d7402a0ad26fc86500ceaa1172d"
server_key_iv: "c7114cab9ce57b0158d4728f4eb2dfc1"
server_key_version: "encryptionkey@0"
I20260807 03:09:08.602188 22700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260807 03:09:08.605691 22715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:08.607021 22700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:09:08.607195 22700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "b7d47daa49da43a3b5bd7d4c53113199"
format_stamp: "Formatted at 2026-08-07 03:09:08 on dist-test-slave-tm1g"
server_key: "690c5d7402a0ad26fc86500ceaa1172d"
server_key_iv: "c7114cab9ce57b0158d4728f4eb2dfc1"
server_key_version: "encryptionkey@0"
I20260807 03:09:08.607434 22700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:08.612998 22700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:08.613844 22700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:08.614122 22700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:08.615140 22700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:08.616658 22700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:08.616791 22700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:08.616915 22700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:08.617010 22700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:08.650264 22700 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:42983
I20260807 03:09:08.650381 22833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:42983 every 8 connection(s)
I20260807 03:09:08.651605 22700 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:09:08.654373 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 22700
I20260807 03:09:08.654582 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:09:08.655071 17532 external_mini_cluster.cc:1428] Setting key 4326775e288a870cd6ac7a26c08b3d07
I20260807 03:09:08.659994 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.654099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:40367 (local address 127.17.31.62:35107)
0807 03:09:08.654354 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:08.654363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:08.656013 (+ 1650us) server_negotiation.cc:407] Connection header received
0807 03:09:08.658138 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:08.658142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:08.658190 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:08.658270 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:08.659430 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:08.659434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:08.659437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:08.659452 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:08.659482 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:08.659493 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:08.659598 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:08.659755 (+ 157us) server_negotiation.cc:299] Negotiation successful
0807 03:09:08.659799 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 03:09:08.660856 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.654766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35107 (local address 127.17.31.3:40367)
0807 03:09:08.655817 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:08.655885 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:08.657863 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:08.658494 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:08.658507 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:08.658591 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:08.659164 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:09:08.659167 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:08.659201 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:09:08.659277 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:08.659579 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:08.659599 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:08.659604 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 03:09:08.659792 (+ 188us) client_negotiation.cc:240] Negotiation successful
0807 03:09:08.659960 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":318,"threads_started":1}
I20260807 03:09:08.662123 22834 heartbeater.cc:347] Connected to a master server at 127.17.31.62:35107
I20260807 03:09:08.662423 22834 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:08.662909 22834 heartbeater.cc:511] Master 127.17.31.62:35107 requested a full tablet report, sending...
I20260807 03:09:08.664135 22366 ts_manager.cc:194] Registered new tserver with Master: b7d47daa49da43a3b5bd7d4c53113199 (127.17.31.3:42983)
I20260807 03:09:08.670078 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:09:08.677366 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.674978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.17.31.62:35107)
0807 03:09:08.675158 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:08.675162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:08.675702 (+ 540us) server_negotiation.cc:407] Connection header received
0807 03:09:08.675810 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:08.675814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:08.675865 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:08.675949 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:08.676856 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:08.676860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:08.676863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:08.676878 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:08.676907 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:08.676918 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:08.677024 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:08.677113 (+ 89us) server_negotiation.cc:299] Negotiation successful
0807 03:09:08.677159 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 03:09:08.681195 22366 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:09:08.688081 22366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:09:08.714687 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.700588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.17.31.2:42023)
0807 03:09:08.710158 (+ 9570us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:08.710172 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:08.710208 (+ 36us) server_negotiation.cc:407] Connection header received
0807 03:09:08.710376 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:08.710389 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:08.710690 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:08.712244 (+ 1554us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:08.713754 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:08.713759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:08.713768 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:08.713786 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:08.713851 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:08.713872 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:08.714084 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:08.714160 (+ 76us) server_negotiation.cc:299] Negotiation successful
0807 03:09:08.714271 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9350,"thread_start_us":9553,"threads_started":1}
I20260807 03:09:08.715090 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.700001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:42983 (local address 127.0.0.1:50770)
0807 03:09:08.701629 (+ 1628us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:08.701688 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:08.701837 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:08.713010 (+ 11173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:08.713018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:08.713047 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:08.713688 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:09:08.713693 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:08.713711 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:09:08.713754 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:08.714279 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:08.714283 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:08.714288 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 03:09:08.714468 (+ 180us) client_negotiation.cc:240] Negotiation successful
0807 03:09:08.714526 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1275,"thread_start_us":1107,"threads_started":1}
I20260807 03:09:08.715135 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.700307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:42023 (local address 127.0.0.1:33826)
0807 03:09:08.709883 (+ 9576us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:08.709908 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:08.710039 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:08.710944 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:08.710955 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:08.711052 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:08.711956 (+ 904us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:09:08.711962 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:08.711993 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:09:08.712084 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:08.714025 (+ 1941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:08.714032 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:08.714037 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 03:09:08.714163 (+ 126us) client_negotiation.cc:240] Negotiation successful
0807 03:09:08.714231 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9403,"mutex_wait_us":24,"thread_start_us":10573,"threads_started":1}
I20260807 03:09:08.715739 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.702599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.17.31.1:41805)
0807 03:09:08.710828 (+ 8229us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:08.710838 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:08.712167 (+ 1329us) server_negotiation.cc:407] Connection header received
0807 03:09:08.712251 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:08.712260 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:08.712492 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:08.712644 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:08.713662 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:08.713671 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:08.713686 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:08.713713 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:08.713768 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:08.713789 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:08.713917 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:08.715422 (+ 1505us) server_negotiation.cc:299] Negotiation successful
0807 03:09:08.715518 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8044,"thread_start_us":7916,"threads_started":1}
I20260807 03:09:08.715794 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.700842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50770 (local address 127.17.31.3:42983)
0807 03:09:08.711335 (+ 10493us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:08.711343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:08.711374 (+ 31us) server_negotiation.cc:407] Connection header received
0807 03:09:08.711450 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:08.711458 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:08.711648 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:08.711815 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:08.713931 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:08.713940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:08.713954 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:08.713975 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:08.714033 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:08.714058 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:08.714217 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:08.715048 (+ 831us) server_negotiation.cc:299] Negotiation successful
0807 03:09:08.715147 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10318,"thread_start_us":10208,"threads_started":1}
I20260807 03:09:08.714738 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.702352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:41805 (local address 127.0.0.1:60652)
0807 03:09:08.711937 (+ 9585us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:08.711955 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:08.712048 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:08.712674 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:08.712682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:08.712710 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:08.713406 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:09:08.713409 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:08.713427 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:09:08.713481 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:08.713906 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:08.713916 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:08.713923 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 03:09:08.714105 (+ 182us) client_negotiation.cc:240] Negotiation successful
0807 03:09:08.714218 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9407,"thread_start_us":7157,"threads_started":1}
I20260807 03:09:08.719012 22630 tablet_service.cc:1467] Processing CreateTablet for tablet fab2e3815cf54262837a5af80b055f1a (DEFAULT_TABLE table=test-table [id=7309d5debeca46a5b0474c6e90afa5c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:08.719262 22493 tablet_service.cc:1467] Processing CreateTablet for tablet fab2e3815cf54262837a5af80b055f1a (DEFAULT_TABLE table=test-table [id=7309d5debeca46a5b0474c6e90afa5c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:08.719262 22766 tablet_service.cc:1467] Processing CreateTablet for tablet fab2e3815cf54262837a5af80b055f1a (DEFAULT_TABLE table=test-table [id=7309d5debeca46a5b0474c6e90afa5c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:08.720302 22630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fab2e3815cf54262837a5af80b055f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:08.720585 22493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fab2e3815cf54262837a5af80b055f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:08.720583 22766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fab2e3815cf54262837a5af80b055f1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:08.732169 22853 tablet_bootstrap.cc:492] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487: Bootstrap starting.
I20260807 03:09:08.734822 22854 tablet_bootstrap.cc:492] T fab2e3815cf54262837a5af80b055f1a P ade6cf5477a54fa5a940f17587aa6942: Bootstrap starting.
I20260807 03:09:08.735625 22853 tablet_bootstrap.cc:654] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:08.736980 22853 log.cc:824] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:08.738382 22854 tablet_bootstrap.cc:654] T fab2e3815cf54262837a5af80b055f1a P ade6cf5477a54fa5a940f17587aa6942: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:08.739384 22855 tablet_bootstrap.cc:492] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199: Bootstrap starting.
I20260807 03:09:08.739610 22854 log.cc:824] T fab2e3815cf54262837a5af80b055f1a P ade6cf5477a54fa5a940f17587aa6942: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:08.740500 22853 tablet_bootstrap.cc:492] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487: No bootstrap required, opened a new log
I20260807 03:09:08.740773 22853 ts_tablet_manager.cc:1397] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 03:09:08.742084 22854 tablet_bootstrap.cc:492] T fab2e3815cf54262837a5af80b055f1a P ade6cf5477a54fa5a940f17587aa6942: No bootstrap required, opened a new log
I20260807 03:09:08.742403 22854 ts_tablet_manager.cc:1397] T fab2e3815cf54262837a5af80b055f1a P ade6cf5477a54fa5a940f17587aa6942: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260807 03:09:08.742962 22855 tablet_bootstrap.cc:654] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:08.744203 22855 log.cc:824] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:08.747265 22855 tablet_bootstrap.cc:492] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199: No bootstrap required, opened a new log
I20260807 03:09:08.747529 22855 ts_tablet_manager.cc:1397] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 03:09:08.750969 22854 raft_consensus.cc:348] T fab2e3815cf54262837a5af80b055f1a P ade6cf5477a54fa5a940f17587aa6942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d47daa49da43a3b5bd7d4c53113199" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42983 } } peers { permanent_uuid: "33ac7808a2104ebcbb8a46ec8186c487" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 42023 } } peers { permanent_uuid: "ade6cf5477a54fa5a940f17587aa6942" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 41805 } }
I20260807 03:09:08.751333 22854 raft_consensus.cc:374] T fab2e3815cf54262837a5af80b055f1a P ade6cf5477a54fa5a940f17587aa6942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:08.751142 22853 raft_consensus.cc:348] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d47daa49da43a3b5bd7d4c53113199" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42983 } } peers { permanent_uuid: "33ac7808a2104ebcbb8a46ec8186c487" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 42023 } } peers { permanent_uuid: "ade6cf5477a54fa5a940f17587aa6942" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 41805 } }
I20260807 03:09:08.751433 22854 raft_consensus.cc:729] T fab2e3815cf54262837a5af80b055f1a P ade6cf5477a54fa5a940f17587aa6942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade6cf5477a54fa5a940f17587aa6942, State: Initialized, Role: FOLLOWER
I20260807 03:09:08.751565 22853 raft_consensus.cc:374] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:08.751662 22853 raft_consensus.cc:729] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33ac7808a2104ebcbb8a46ec8186c487, State: Initialized, Role: FOLLOWER
I20260807 03:09:08.752290 22854 consensus_queue.cc:260] T fab2e3815cf54262837a5af80b055f1a P ade6cf5477a54fa5a940f17587aa6942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d47daa49da43a3b5bd7d4c53113199" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42983 } } peers { permanent_uuid: "33ac7808a2104ebcbb8a46ec8186c487" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 42023 } } peers { permanent_uuid: "ade6cf5477a54fa5a940f17587aa6942" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 41805 } }
I20260807 03:09:08.752342 22853 consensus_queue.cc:260] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d47daa49da43a3b5bd7d4c53113199" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42983 } } peers { permanent_uuid: "33ac7808a2104ebcbb8a46ec8186c487" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 42023 } } peers { permanent_uuid: "ade6cf5477a54fa5a940f17587aa6942" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 41805 } }
I20260807 03:09:08.753892 22560 heartbeater.cc:503] Master 127.17.31.62:35107 was elected leader, sending a full tablet report...
I20260807 03:09:08.753882 22697 heartbeater.cc:503] Master 127.17.31.62:35107 was elected leader, sending a full tablet report...
I20260807 03:09:08.754460 22853 ts_tablet_manager.cc:1428] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260807 03:09:08.755702 22854 ts_tablet_manager.cc:1428] T fab2e3815cf54262837a5af80b055f1a P ade6cf5477a54fa5a940f17587aa6942: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260807 03:09:08.757541 22855 raft_consensus.cc:348] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d47daa49da43a3b5bd7d4c53113199" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42983 } } peers { permanent_uuid: "33ac7808a2104ebcbb8a46ec8186c487" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 42023 } } peers { permanent_uuid: "ade6cf5477a54fa5a940f17587aa6942" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 41805 } }
I20260807 03:09:08.757962 22855 raft_consensus.cc:374] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:08.758069 22855 raft_consensus.cc:729] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7d47daa49da43a3b5bd7d4c53113199, State: Initialized, Role: FOLLOWER
I20260807 03:09:08.758684 22855 consensus_queue.cc:260] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d47daa49da43a3b5bd7d4c53113199" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42983 } } peers { permanent_uuid: "33ac7808a2104ebcbb8a46ec8186c487" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 42023 } } peers { permanent_uuid: "ade6cf5477a54fa5a940f17587aa6942" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 41805 } }
I20260807 03:09:08.759740 22834 heartbeater.cc:503] Master 127.17.31.62:35107 was elected leader, sending a full tablet report...
I20260807 03:09:08.760321 22855 ts_tablet_manager.cc:1428] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260807 03:09:08.787448 22561 tablet.cc:2369] T fab2e3815cf54262837a5af80b055f1a P ade6cf5477a54fa5a940f17587aa6942: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:09:08.878935 22860 raft_consensus.cc:482] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:09:08.879279 22860 raft_consensus.cc:504] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d47daa49da43a3b5bd7d4c53113199" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42983 } } peers { permanent_uuid: "33ac7808a2104ebcbb8a46ec8186c487" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 42023 } } peers { permanent_uuid: "ade6cf5477a54fa5a940f17587aa6942" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 41805 } }
I20260807 03:09:08.880988 22860 leader_election.cc:288] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7d47daa49da43a3b5bd7d4c53113199 (127.17.31.3:42983), ade6cf5477a54fa5a940f17587aa6942 (127.17.31.1:41805)
I20260807 03:09:08.884665 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.881512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:43039 (local address 127.17.31.1:41805)
0807 03:09:08.881700 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:08.881706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:08.882540 (+ 834us) server_negotiation.cc:407] Connection header received
0807 03:09:08.882760 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:08.882766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:08.882829 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:08.882943 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:08.884027 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:08.884032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:08.884036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:08.884055 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:08.884095 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:08.884110 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:08.884232 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:08.884348 (+ 116us) server_negotiation.cc:299] Negotiation successful
0807 03:09:08.884401 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 03:09:08.884665 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.881611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:51041 (local address 127.17.31.3:42983)
0807 03:09:08.881781 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:08.881785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:08.882924 (+ 1139us) server_negotiation.cc:407] Connection header received
0807 03:09:08.883064 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:08.883069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:08.883124 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:08.883198 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:08.884048 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:08.884053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:08.884057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:08.884075 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:08.884110 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:08.884125 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:08.884238 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:08.884335 (+ 97us) server_negotiation.cc:299] Negotiation successful
0807 03:09:08.884398 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 03:09:08.885113 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.881562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:41805 (local address 127.17.31.2:43039)
0807 03:09:08.882410 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:08.882430 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:08.882604 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:08.883000 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:08.883015 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:08.883043 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:08.883771 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:09:08.883776 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:08.883796 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:09:08.883854 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:08.884221 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:08.884226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:08.884231 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 03:09:08.884427 (+ 196us) client_negotiation.cc:240] Negotiation successful
0807 03:09:08.884494 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":21,"thread_start_us":947,"threads_started":1}
I20260807 03:09:08.885135 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.881591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:42983 (local address 127.17.31.2:51041)
0807 03:09:08.882804 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:08.882818 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:08.882939 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:08.883266 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:08.883271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:08.883296 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:08.883852 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:09:08.883855 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:08.883867 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:09:08.883899 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:08.884221 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:08.884226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:08.884231 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 03:09:08.884416 (+ 185us) client_negotiation.cc:240] Negotiation successful
0807 03:09:08.884482 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1121,"spinlock_wait_cycles":28928,"thread_start_us":434,"threads_started":1}
I20260807 03:09:08.886111 22786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fab2e3815cf54262837a5af80b055f1a" candidate_uuid: "33ac7808a2104ebcbb8a46ec8186c487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d47daa49da43a3b5bd7d4c53113199" is_pre_election: true
I20260807 03:09:08.886201 22513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fab2e3815cf54262837a5af80b055f1a" candidate_uuid: "33ac7808a2104ebcbb8a46ec8186c487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade6cf5477a54fa5a940f17587aa6942" is_pre_election: true
I20260807 03:09:08.886649 22786 raft_consensus.cc:2454] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33ac7808a2104ebcbb8a46ec8186c487 in term 0.
I20260807 03:09:08.886763 22513 raft_consensus.cc:2454] T fab2e3815cf54262837a5af80b055f1a P ade6cf5477a54fa5a940f17587aa6942 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33ac7808a2104ebcbb8a46ec8186c487 in term 0.
I20260807 03:09:08.887288 22583 leader_election.cc:302] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33ac7808a2104ebcbb8a46ec8186c487, b7d47daa49da43a3b5bd7d4c53113199; no voters:
I20260807 03:09:08.887661 22860 raft_consensus.cc:2781] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:09:08.887811 22860 raft_consensus.cc:482] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:09:08.887925 22860 raft_consensus.cc:3037] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:08.890344 22860 raft_consensus.cc:504] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d47daa49da43a3b5bd7d4c53113199" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42983 } } peers { permanent_uuid: "33ac7808a2104ebcbb8a46ec8186c487" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 42023 } } peers { permanent_uuid: "ade6cf5477a54fa5a940f17587aa6942" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 41805 } }
I20260807 03:09:08.890910 22860 leader_election.cc:288] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [CANDIDATE]: Term 1 election: Requested vote from peers b7d47daa49da43a3b5bd7d4c53113199 (127.17.31.3:42983), ade6cf5477a54fa5a940f17587aa6942 (127.17.31.1:41805)
I20260807 03:09:08.891376 22513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fab2e3815cf54262837a5af80b055f1a" candidate_uuid: "33ac7808a2104ebcbb8a46ec8186c487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ade6cf5477a54fa5a940f17587aa6942"
I20260807 03:09:08.891376 22786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fab2e3815cf54262837a5af80b055f1a" candidate_uuid: "33ac7808a2104ebcbb8a46ec8186c487" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7d47daa49da43a3b5bd7d4c53113199"
I20260807 03:09:08.891585 22513 raft_consensus.cc:3037] T fab2e3815cf54262837a5af80b055f1a P ade6cf5477a54fa5a940f17587aa6942 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:08.891589 22786 raft_consensus.cc:3037] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:08.894825 22786 raft_consensus.cc:2454] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33ac7808a2104ebcbb8a46ec8186c487 in term 1.
I20260807 03:09:08.894830 22513 raft_consensus.cc:2454] T fab2e3815cf54262837a5af80b055f1a P ade6cf5477a54fa5a940f17587aa6942 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33ac7808a2104ebcbb8a46ec8186c487 in term 1.
I20260807 03:09:08.895259 22584 leader_election.cc:302] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 33ac7808a2104ebcbb8a46ec8186c487, ade6cf5477a54fa5a940f17587aa6942, b7d47daa49da43a3b5bd7d4c53113199; no voters:
I20260807 03:09:08.895625 22860 raft_consensus.cc:2781] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:08.896080 22860 raft_consensus.cc:686] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [term 1 LEADER]: Becoming Leader. State: Replica: 33ac7808a2104ebcbb8a46ec8186c487, State: Running, Role: LEADER
I20260807 03:09:08.896544 22860 consensus_queue.cc:237] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d47daa49da43a3b5bd7d4c53113199" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42983 } } peers { permanent_uuid: "33ac7808a2104ebcbb8a46ec8186c487" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 42023 } } peers { permanent_uuid: "ade6cf5477a54fa5a940f17587aa6942" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 41805 } }
I20260807 03:09:08.900825 22366 catalog_manager.cc:5495] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33ac7808a2104ebcbb8a46ec8186c487 (127.17.31.2). New cstate: current_term: 1 leader_uuid: "33ac7808a2104ebcbb8a46ec8186c487" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d47daa49da43a3b5bd7d4c53113199" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33ac7808a2104ebcbb8a46ec8186c487" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 42023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ade6cf5477a54fa5a940f17587aa6942" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 41805 } health_report { overall_health: UNKNOWN } } }
W20260807 03:09:08.903228 22835 tablet.cc:2369] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:09:08.943573 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.941567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33842 (local address 127.17.31.2:42023)
0807 03:09:08.941734 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:08.941738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:08.941755 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:09:08.941939 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:08.941943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:08.942026 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:08.942107 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:08.943014 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:08.943018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:08.943021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:08.943035 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:08.943067 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:08.943079 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:08.943208 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:08.943272 (+ 64us) server_negotiation.cc:299] Negotiation successful
0807 03:09:08.943360 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 03:09:08.951439 22513 raft_consensus.cc:1261] T fab2e3815cf54262837a5af80b055f1a P ade6cf5477a54fa5a940f17587aa6942 [term 1 FOLLOWER]: Refusing update from remote peer 33ac7808a2104ebcbb8a46ec8186c487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:09:08.951474 22786 raft_consensus.cc:1261] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199 [term 1 FOLLOWER]: Refusing update from remote peer 33ac7808a2104ebcbb8a46ec8186c487: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:09:08.952459 22865 consensus_queue.cc:1035] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7d47daa49da43a3b5bd7d4c53113199" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 42983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:08.952800 22860 consensus_queue.cc:1035] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ade6cf5477a54fa5a940f17587aa6942" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:08.962368 22868 mvcc.cc:205] Tried to move back new op lower bound from 7315751522095898624 to 7315751521883811840. Current Snapshot: MvccSnapshot[applied={T|T < 7315751522095898624}]
I20260807 03:09:08.965060 22869 mvcc.cc:205] Tried to move back new op lower bound from 7315751522095898624 to 7315751521883811840. Current Snapshot: MvccSnapshot[applied={T|T < 7315751522095898624}]
I20260807 03:09:08.980240 22366 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:09:08.980541 22366 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:09:08.984150 22366 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c053f9d52ae74b2f8352f47f2fb1f18a: Sending DeleteTablet for 3 replicas of tablet fab2e3815cf54262837a5af80b055f1a
I20260807 03:09:08.985239 22766 tablet_service.cc:1514] Processing DeleteTablet for tablet fab2e3815cf54262837a5af80b055f1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:08 UTC) from {username='slave'} at 127.0.0.1:50770
I20260807 03:09:08.985249 22630 tablet_service.cc:1514] Processing DeleteTablet for tablet fab2e3815cf54262837a5af80b055f1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:08 UTC) from {username='slave'} at 127.0.0.1:33826
I20260807 03:09:08.985325 22493 tablet_service.cc:1514] Processing DeleteTablet for tablet fab2e3815cf54262837a5af80b055f1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:08 UTC) from {username='slave'} at 127.0.0.1:60652
I20260807 03:09:08.993389 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 22427
I20260807 03:09:09.000041 22877 tablet_replica.cc:330] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199: stopping tablet replica
I20260807 03:09:09.000566 22877 raft_consensus.cc:2229] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:09.001190 22878 tablet_replica.cc:330] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487: stopping tablet replica
I20260807 03:09:09.001210 22877 raft_consensus.cc:2258] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:09.001600 22878 raft_consensus.cc:2229] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:09:09.002200 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:08.996590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60660 (local address 127.17.31.1:41805)
0807 03:09:08.996809 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:08.996818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:08.996841 (+ 23us) server_negotiation.cc:407] Connection header received
0807 03:09:08.996911 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:08.996919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:08.997061 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:08.997168 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:09.000338 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:09.000344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:09.000348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:09.000368 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:09.000432 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:09.000453 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:09.000626 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:09.001760 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0807 03:09:09.001830 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 03:09:09.003458 22878 raft_consensus.cc:2258] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:09.003722 22555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:09.003742 22877 ts_tablet_manager.cc:1905] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:09.011188 22878 ts_tablet_manager.cc:1905] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:09.013198 22877 ts_tablet_manager.cc:1918] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:09:09.013375 22877 log.cc:1196] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/fab2e3815cf54262837a5af80b055f1a
I20260807 03:09:09.014026 22877 ts_tablet_manager.cc:1939] T fab2e3815cf54262837a5af80b055f1a P b7d47daa49da43a3b5bd7d4c53113199: Deleting consensus metadata
I20260807 03:09:09.016340 22351 catalog_manager.cc:4841] TS b7d47daa49da43a3b5bd7d4c53113199 (127.17.31.3:42983): tablet fab2e3815cf54262837a5af80b055f1a (table test-table [id=7309d5debeca46a5b0474c6e90afa5c0]) successfully deleted
I20260807 03:09:09.023600 22878 ts_tablet_manager.cc:1918] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:09:09.023779 22878 log.cc:1196] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/fab2e3815cf54262837a5af80b055f1a
I20260807 03:09:09.024346 22878 ts_tablet_manager.cc:1939] T fab2e3815cf54262837a5af80b055f1a P 33ac7808a2104ebcbb8a46ec8186c487: Deleting consensus metadata
I20260807 03:09:09.026185 22350 catalog_manager.cc:4841] TS 33ac7808a2104ebcbb8a46ec8186c487 (127.17.31.2:42023): tablet fab2e3815cf54262837a5af80b055f1a (table test-table [id=7309d5debeca46a5b0474c6e90afa5c0]) successfully deleted
I20260807 03:09:09.108371 22879 tablet_replica.cc:330] T fab2e3815cf54262837a5af80b055f1a P ade6cf5477a54fa5a940f17587aa6942: stopping tablet replica
I20260807 03:09:09.108687 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 22427
W20260807 03:09:09.132686 22352 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.31.1:41805 (error 108)
I20260807 03:09:09.133452 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 22564
I20260807 03:09:09.134861 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:09.134288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:41805 (local address 127.0.0.1:60666)
0807 03:09:09.134560 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:09.134691 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260807 03:09:09.135115 22352 catalog_manager.cc:4584] TS ade6cf5477a54fa5a940f17587aa6942 (127.17.31.1:41805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fab2e3815cf54262837a5af80b055f1a: Network error: Client connection negotiation failed: client connection to 127.17.31.1:41805: connect: Connection refused (error 111)
I20260807 03:09:09.136763 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:09.134259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.17.31.2:42023)
0807 03:09:09.134460 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:09.134466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:09.134602 (+ 136us) server_negotiation.cc:407] Connection header received
0807 03:09:09.134850 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:09.134854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:09.134905 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:09.135022 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:09.136117 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:09.136127 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:09.136130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:09.136146 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:09.136177 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:09.136189 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:09.136332 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:09.136429 (+ 97us) server_negotiation.cc:299] Negotiation successful
0807 03:09:09.136485 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 03:09:09.137488 22692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:09.188742 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:09.188291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:41805 (local address 127.0.0.1:60670)
0807 03:09:09.188513 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:09.188592 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 03:09:09.189736 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:09.189394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:41805 (local address 127.0.0.1:60680)
0807 03:09:09.189561 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:09.189630 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 03:09:09.246932 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 22564
I20260807 03:09:09.250279 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:09.249689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:41805 (local address 127.0.0.1:60684)
0807 03:09:09.249944 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:09.250060 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260807 03:09:09.251390 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:09.250964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:41805 (local address 127.0.0.1:60694)
0807 03:09:09.251143 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:09.251233 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 03:09:09.266712 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 22700
I20260807 03:09:09.269668 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:09.267662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.17.31.3:42983)
0807 03:09:09.267872 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:09.267876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:09.267890 (+ 14us) server_negotiation.cc:407] Connection header received
0807 03:09:09.267941 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:09.267945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:09.268034 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:09.268147 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:09.269053 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:09.269058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:09.269064 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:09.269086 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:09.269115 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:09.269127 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:09.269261 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:09.269348 (+ 87us) server_negotiation.cc:299] Negotiation successful
0807 03:09:09.269406 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 03:09:09.270596 22829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:09.331735 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:09.331270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:41805 (local address 127.0.0.1:60710)
0807 03:09:09.331474 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:09.331574 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260807 03:09:09.332844 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:09.332467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:41805 (local address 127.0.0.1:60724)
0807 03:09:09.332668 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:09.332733 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:41805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260807 03:09:09.364470 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 22700
I20260807 03:09:09.382095 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 22331
I20260807 03:09:09.383325 22398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:09.488207 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 22331
2026-08-07T03:09:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2703 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260807 03:09:09.531284 17532 mini_oidc.cc:152] Starting JWKS server
I20260807 03:09:09.534116 17532 webserver.cc:460] Webserver started at https://127.0.0.1:43219/ using document root <none> and password file <none>
I20260807 03:09:09.534471 17532 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 03:09:09.535121 17532 webserver.cc:460] Webserver started at http://127.0.0.1:33111/ using document root <none> and password file <none>
2026-08-07T03:09:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:09:09Z Disabled control of system clock
I20260807 03:09:09.563196 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:39143
--webserver_interface=127.17.31.62
--webserver_port=0
--builtin_ntp_servers=127.17.31.20:45995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:39143
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43219/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/testchainca.cert with env {}
W20260807 03:09:09.885068 22891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:09.885416 22891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:09.885583 22891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:09.893640 22891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:09:09.893798 22891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:09.893872 22891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:09.893956 22891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:09:09.894024 22891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 03:09:09.894119 22891 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 03:09:09.894193 22891 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43219/jwks.json
I20260807 03:09:09.904222 22891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:45995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:39143
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:39143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43219/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:09.905534 22891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:09.906880 22891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:09.918794 22899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:09.918843 22897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:09.918867 22896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:09.919756 22891 server_base.cc:1034] running on GCE node
I20260807 03:09:09.920463 22891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:09.921736 22891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:09.922960 22891 hybrid_clock.cc:648] HybridClock initialized: now 1786072149922915 us; error 51 us; skew 500 ppm
I20260807 03:09:09.923291 22891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:09:09.925464 22891 webserver.cc:460] Webserver started at http://127.17.31.62:37235/ using document root <none> and password file <none>
I20260807 03:09:09.926067 22891 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:09.926154 22891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:09.926488 22891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:09.929311 22891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "c82a4ede9b0244939c718769e00d3d53"
format_stamp: "Formatted at 2026-08-07 03:09:09 on dist-test-slave-tm1g"
server_key: "c9c5cb8e9937ebaf98e30c231862774e"
server_key_iv: "1107856fb9c646b5f93d5e1ba83abe90"
server_key_version: "encryptionkey@0"
I20260807 03:09:09.929998 22891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "c82a4ede9b0244939c718769e00d3d53"
format_stamp: "Formatted at 2026-08-07 03:09:09 on dist-test-slave-tm1g"
server_key: "c9c5cb8e9937ebaf98e30c231862774e"
server_key_iv: "1107856fb9c646b5f93d5e1ba83abe90"
server_key_version: "encryptionkey@0"
I20260807 03:09:09.934751 22891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260807 03:09:09.937822 22906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:09.939208 22891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:09:09.939337 22891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "c82a4ede9b0244939c718769e00d3d53"
format_stamp: "Formatted at 2026-08-07 03:09:09 on dist-test-slave-tm1g"
server_key: "c9c5cb8e9937ebaf98e30c231862774e"
server_key_iv: "1107856fb9c646b5f93d5e1ba83abe90"
server_key_version: "encryptionkey@0"
I20260807 03:09:09.939492 22891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:09.960461 22891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:09.961133 22891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:09.961397 22891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:09.979305 22891 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:39143
I20260807 03:09:09.979336 22961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:39143 every 8 connection(s)
I20260807 03:09:09.980528 22891 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:09:09.983903 22962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:09.989236 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 22891
I20260807 03:09:09.989439 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:09:09.990254 17532 external_mini_cluster.cc:1428] Setting key e3efe1a4b31dc185b2c9260932485d64
I20260807 03:09:09.994697 22962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53: Bootstrap starting.
I20260807 03:09:09.997990 22962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:09.999207 22962 log.cc:824] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:10.002637 22962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53: No bootstrap required, opened a new log
I20260807 03:09:10.005069 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:09.992205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.17.31.62:39143)
0807 03:09:09.993181 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:09.993192 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:09.993229 (+ 37us) server_negotiation.cc:407] Connection header received
0807 03:09:09.994919 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:09.994941 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:09.995352 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:09.995648 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:09.996322 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:09.997545 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:09.999524 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:10.000653 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:10.000796 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:10.002181 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:10.002192 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:10.002208 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:10.002984 (+ 776us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:10.003093 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:10.003123 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:10.003239 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:10.003601 (+ 362us) server_negotiation.cc:299] Negotiation successful
0807 03:09:10.003890 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":388,"threads_started":1}
I20260807 03:09:10.008759 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:09:10.011816 22962 raft_consensus.cc:348] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82a4ede9b0244939c718769e00d3d53" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39143 } }
I20260807 03:09:10.012145 22962 raft_consensus.cc:374] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:10.012246 22962 raft_consensus.cc:729] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82a4ede9b0244939c718769e00d3d53, State: Initialized, Role: FOLLOWER
I20260807 03:09:10.012820 22962 consensus_queue.cc:260] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82a4ede9b0244939c718769e00d3d53" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39143 } }
I20260807 03:09:10.013038 22962 raft_consensus.cc:388] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:09:10.013159 22962 raft_consensus.cc:482] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:09:10.013340 22962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:10.015415 22962 raft_consensus.cc:504] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82a4ede9b0244939c718769e00d3d53" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39143 } }
I20260807 03:09:10.015869 22962 leader_election.cc:302] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c82a4ede9b0244939c718769e00d3d53; no voters:
I20260807 03:09:10.016446 22962 leader_election.cc:288] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:09:10.016633 22967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:10.017539 22967 raft_consensus.cc:686] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [term 1 LEADER]: Becoming Leader. State: Replica: c82a4ede9b0244939c718769e00d3d53, State: Running, Role: LEADER
I20260807 03:09:10.018029 22967 consensus_queue.cc:237] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82a4ede9b0244939c718769e00d3d53" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39143 } }
I20260807 03:09:10.018548 22962 sys_catalog.cc:564] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:09:10.020892 22968 sys_catalog.cc:455] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c82a4ede9b0244939c718769e00d3d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82a4ede9b0244939c718769e00d3d53" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39143 } } }
I20260807 03:09:10.021339 22968 sys_catalog.cc:458] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:10.022090 22969 sys_catalog.cc:455] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c82a4ede9b0244939c718769e00d3d53. Latest consensus state: current_term: 1 leader_uuid: "c82a4ede9b0244939c718769e00d3d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c82a4ede9b0244939c718769e00d3d53" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39143 } } }
I20260807 03:09:10.022645 22969 sys_catalog.cc:458] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53 [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:10.023317 22975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:09:10.028366 22975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:09:10.039559 22975 catalog_manager.cc:1347] Generated new cluster ID: a6f748fc8c5b41eea296f331c027acf9
I20260807 03:09:10.039700 22975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:09:10.064953 22975 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:09:10.066156 22975 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:09:10.082813 22975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c82a4ede9b0244939c718769e00d3d53: Generated new TSK 0
I20260807 03:09:10.083561 22975 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 03:09:10.133865 22992 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260807 03:09:10.134070 22992 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 03:09:10.135025 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:10.106651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.17.31.62:39143)
0807 03:09:10.106846 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:10.106854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:10.107256 (+ 402us) server_negotiation.cc:407] Connection header received
0807 03:09:10.107657 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:10.107661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:10.107731 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:10.107887 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0807 03:09:10.108453 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:10.109182 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:10.110719 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:10.111200 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:10.111356 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:10.111867 (+ 511us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 03:09:10.134376 (+ 22509us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43219/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 03:09:10.134731 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41650: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43219/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260807 03:09:10.135165 22965 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41650: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43219/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260807 03:09:10.137015 22991 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:10.106545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:39143 (local address 127.0.0.1:41650)
0807 03:09:10.107148 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:10.107164 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:10.107356 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:10.107914 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:10.107920 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:10.107934 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0807 03:09:10.108302 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:10.108311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:10.109339 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:10.109345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:10.110573 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:10.110584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:10.111390 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:10.111396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:10.111553 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:10.111609 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 03:09:10.134668 (+ 23059us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43219/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 03:09:10.135026 (+ 358us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.31.62:39143: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43219/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":319,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.17.31.62:39143: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43219/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260807 03:09:10.138707 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 22891
I20260807 03:09:10.139796 22957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:10.233527 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 22891
2026-08-07T03:09:10Z chronyd exiting
I20260807 03:09:10.285287 17532 test_util.cc:175] -----------------------------------------------
I20260807 03:09:10.285421 17532 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786072106928531-17532-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (758 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260807 03:09:10.289188 17532 mini_oidc.cc:152] Starting JWKS server
I20260807 03:09:10.291023 17532 webserver.cc:460] Webserver started at https://127.0.0.1:41017/ using document root <none> and password file <none>
I20260807 03:09:10.291239 17532 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 03:09:10.291826 17532 webserver.cc:460] Webserver started at http://127.0.0.1:37455/ using document root <none> and password file <none>
2026-08-07T03:09:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:09:10Z Disabled control of system clock
I20260807 03:09:10.320032 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:36241
--webserver_interface=127.17.31.62
--webserver_port=0
--builtin_ntp_servers=127.17.31.20:43393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:36241
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41017/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/testchainca.cert with env {}
W20260807 03:09:10.633033 22999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:10.633345 22999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:10.633523 22999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:10.640540 22999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:09:10.640672 22999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:10.640761 22999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:10.640851 22999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:09:10.640918 22999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 03:09:10.641004 22999 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 03:09:10.641088 22999 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41017/jwks.json
I20260807 03:09:10.650965 22999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:43393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:36241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:36241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41017/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:10.652145 22999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:10.653388 22999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:10.665014 23004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:10.665091 23009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:10.665048 23005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:10.666159 22999 server_base.cc:1034] running on GCE node
I20260807 03:09:10.666944 22999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:10.668207 22999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:10.669404 22999 hybrid_clock.cc:648] HybridClock initialized: now 1786072150669353 us; error 40 us; skew 500 ppm
I20260807 03:09:10.669734 22999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:09:10.675930 22999 webserver.cc:460] Webserver started at http://127.17.31.62:37939/ using document root <none> and password file <none>
I20260807 03:09:10.676605 22999 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:10.676736 22999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:10.677073 22999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:10.679870 22999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "129590a35ee74c439aaaf8033ffe8a79"
format_stamp: "Formatted at 2026-08-07 03:09:10 on dist-test-slave-tm1g"
server_key: "bb6cd7c9d6377f4303bfde35be8036cd"
server_key_iv: "b2cdc75ebdf3ab80d72a53888e0b097a"
server_key_version: "encryptionkey@0"
I20260807 03:09:10.680630 22999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "129590a35ee74c439aaaf8033ffe8a79"
format_stamp: "Formatted at 2026-08-07 03:09:10 on dist-test-slave-tm1g"
server_key: "bb6cd7c9d6377f4303bfde35be8036cd"
server_key_iv: "b2cdc75ebdf3ab80d72a53888e0b097a"
server_key_version: "encryptionkey@0"
I20260807 03:09:10.685433 22999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 03:09:10.688624 23016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:10.689832 22999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 03:09:10.690019 22999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "129590a35ee74c439aaaf8033ffe8a79"
format_stamp: "Formatted at 2026-08-07 03:09:10 on dist-test-slave-tm1g"
server_key: "bb6cd7c9d6377f4303bfde35be8036cd"
server_key_iv: "b2cdc75ebdf3ab80d72a53888e0b097a"
server_key_version: "encryptionkey@0"
I20260807 03:09:10.690231 22999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:10.700765 22999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:10.701475 22999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:10.701790 22999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:10.719686 22999 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:36241
I20260807 03:09:10.719707 23069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:36241 every 8 connection(s)
I20260807 03:09:10.720932 22999 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:09:10.724503 23070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:10.725476 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 22999
I20260807 03:09:10.725700 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:09:10.726397 17532 external_mini_cluster.cc:1428] Setting key 9146fde3fc1d55692995f41f94aa1ce7
I20260807 03:09:10.737984 23070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79: Bootstrap starting.
I20260807 03:09:10.739595 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:10.728183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.17.31.62:36241)
0807 03:09:10.729397 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:10.729407 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:10.729451 (+ 44us) server_negotiation.cc:407] Connection header received
0807 03:09:10.730963 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:10.730982 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:10.731416 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:10.731712 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:10.732262 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:10.733475 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:10.735366 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:10.736001 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:10.736139 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:10.737368 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:10.737375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:10.737387 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:10.737870 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:10.737939 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:10.737958 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:10.738066 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:10.738352 (+ 286us) server_negotiation.cc:299] Negotiation successful
0807 03:09:10.738686 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":541,"threads_started":1}
I20260807 03:09:10.741241 23070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:10.742471 23070 log.cc:824] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:10.743285 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:09:10.745501 23070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79: No bootstrap required, opened a new log
I20260807 03:09:10.752506 23070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129590a35ee74c439aaaf8033ffe8a79" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36241 } }
I20260807 03:09:10.752951 23070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:10.753080 23070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129590a35ee74c439aaaf8033ffe8a79, State: Initialized, Role: FOLLOWER
I20260807 03:09:10.753703 23070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129590a35ee74c439aaaf8033ffe8a79" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36241 } }
I20260807 03:09:10.753937 23070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:09:10.754092 23070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:09:10.754254 23070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:10.756299 23070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129590a35ee74c439aaaf8033ffe8a79" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36241 } }
I20260807 03:09:10.756765 23070 leader_election.cc:302] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 129590a35ee74c439aaaf8033ffe8a79; no voters:
I20260807 03:09:10.757309 23070 leader_election.cc:288] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:09:10.757470 23075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:10.758705 23075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [term 1 LEADER]: Becoming Leader. State: Replica: 129590a35ee74c439aaaf8033ffe8a79, State: Running, Role: LEADER
I20260807 03:09:10.759183 23075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129590a35ee74c439aaaf8033ffe8a79" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36241 } }
I20260807 03:09:10.759461 23070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:09:10.761854 23076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "129590a35ee74c439aaaf8033ffe8a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129590a35ee74c439aaaf8033ffe8a79" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36241 } } }
I20260807 03:09:10.762253 23076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:10.763281 23077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 129590a35ee74c439aaaf8033ffe8a79. Latest consensus state: current_term: 1 leader_uuid: "129590a35ee74c439aaaf8033ffe8a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "129590a35ee74c439aaaf8033ffe8a79" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36241 } } }
I20260807 03:09:10.763665 23077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79 [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:10.764999 23082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:09:10.770272 23082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:09:10.778803 23082 catalog_manager.cc:1347] Generated new cluster ID: 2524a38f186f4c7d9e76578ade566d2e
I20260807 03:09:10.778903 23082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:09:10.792480 23082 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:09:10.793756 23082 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:09:10.804000 23082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 129590a35ee74c439aaaf8033ffe8a79: Generated new TSK 0
I20260807 03:09:10.804934 23082 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 03:09:10.839016 23100 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260807 03:09:10.839138 23100 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 03:09:10.840080 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:10.820557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39902 (local address 127.17.31.62:36241)
0807 03:09:10.820788 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:10.820795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:10.821192 (+ 397us) server_negotiation.cc:407] Connection header received
0807 03:09:10.821429 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:10.821433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:10.821498 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:10.821617 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0807 03:09:10.822198 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:10.822948 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:10.824941 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:10.825359 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:10.825446 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:10.826022 (+ 576us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 03:09:10.839579 (+ 13557us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41017/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 03:09:10.839857 (+ 278us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39902: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41017/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":65}
W20260807 03:09:10.840219 23073 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39902: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41017/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260807 03:09:10.840440 23099 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:10.820443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:36241 (local address 127.0.0.1:39902)
0807 03:09:10.821083 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:10.821103 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:10.821265 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:10.821672 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:10.821678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:10.821691 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0807 03:09:10.822019 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:10.822042 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:10.823102 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:10.823108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:10.824795 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:10.824810 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:10.825529 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:10.825536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:10.825735 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:10.825785 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 03:09:10.839786 (+ 14001us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41017/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 03:09:10.840050 (+ 264us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.31.62:36241: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41017/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":331,"threads_started":1}
I20260807 03:09:10.841773 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 22999
I20260807 03:09:10.842926 23065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:10.933271 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 22999
2026-08-07T03:09:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (700 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260807 03:09:10.989295 17532 mini_oidc.cc:152] Starting JWKS server
I20260807 03:09:10.991281 17532 webserver.cc:460] Webserver started at https://127.0.0.1:35567/ using document root <none> and password file <none>
I20260807 03:09:10.991521 17532 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 03:09:10.992175 17532 webserver.cc:460] Webserver started at http://127.0.0.1:35207/ using document root <none> and password file <none>
2026-08-07T03:09:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:09:11Z Disabled control of system clock
I20260807 03:09:11.023901 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:35589
--webserver_interface=127.17.31.62
--webserver_port=0
--builtin_ntp_servers=127.17.31.20:36575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:35589
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35567/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 03:09:11.342146 23107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:11.342518 23107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:11.342669 23107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:11.349771 23107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:09:11.349890 23107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:11.349959 23107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:11.350032 23107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:09:11.350095 23107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 03:09:11.350180 23107 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 03:09:11.350234 23107 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35567/jwks.json
I20260807 03:09:11.359074 23107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:36575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:35589
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:35589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35567/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:11.360309 23107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:11.362078 23107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:11.373558 23115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:11.373603 23113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:11.373626 23112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:11.374644 23107 server_base.cc:1034] running on GCE node
I20260807 03:09:11.375435 23107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:11.376739 23107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:11.377970 23107 hybrid_clock.cc:648] HybridClock initialized: now 1786072151377912 us; error 46 us; skew 500 ppm
I20260807 03:09:11.378353 23107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:09:11.380506 23107 webserver.cc:460] Webserver started at http://127.17.31.62:42445/ using document root <none> and password file <none>
I20260807 03:09:11.381304 23107 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:11.381518 23107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:11.381856 23107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:11.384660 23107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "802427dbce0f4815af411c22e72ec80f"
format_stamp: "Formatted at 2026-08-07 03:09:11 on dist-test-slave-tm1g"
server_key: "529f31d3a1e763ec952d5ed890cff6a0"
server_key_iv: "248c1b0018ea2e2f0b8782799ed92386"
server_key_version: "encryptionkey@0"
I20260807 03:09:11.385471 23107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "802427dbce0f4815af411c22e72ec80f"
format_stamp: "Formatted at 2026-08-07 03:09:11 on dist-test-slave-tm1g"
server_key: "529f31d3a1e763ec952d5ed890cff6a0"
server_key_iv: "248c1b0018ea2e2f0b8782799ed92386"
server_key_version: "encryptionkey@0"
I20260807 03:09:11.390286 23107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260807 03:09:11.393736 23122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:11.395165 23107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 03:09:11.395358 23107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "802427dbce0f4815af411c22e72ec80f"
format_stamp: "Formatted at 2026-08-07 03:09:11 on dist-test-slave-tm1g"
server_key: "529f31d3a1e763ec952d5ed890cff6a0"
server_key_iv: "248c1b0018ea2e2f0b8782799ed92386"
server_key_version: "encryptionkey@0"
I20260807 03:09:11.395576 23107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:11.414825 23107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:11.415589 23107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:11.415896 23107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:11.434051 23107 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:35589
I20260807 03:09:11.434082 23176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:35589 every 8 connection(s)
I20260807 03:09:11.435312 23107 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:09:11.438777 23177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:11.441924 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 23107
I20260807 03:09:11.442127 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:09:11.442984 17532 external_mini_cluster.cc:1428] Setting key 78b51bf98bcd49c6bf0774f2bae5dc8a
I20260807 03:09:11.450475 23177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f: Bootstrap starting.
I20260807 03:09:11.453900 23177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:11.455180 23177 log.cc:824] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:11.457958 23177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f: No bootstrap required, opened a new log
I20260807 03:09:11.458047 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:11.445198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57924 (local address 127.17.31.62:35589)
0807 03:09:11.446367 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:11.446384 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:11.447048 (+ 664us) server_negotiation.cc:407] Connection header received
0807 03:09:11.449047 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:11.449067 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:11.449483 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:11.449788 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:11.450426 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:11.451770 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:11.453599 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:11.454436 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:11.454562 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:11.455857 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:11.455865 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:11.455881 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:11.456398 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:11.456482 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:11.456506 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:11.456647 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:11.456866 (+ 219us) server_negotiation.cc:299] Negotiation successful
0807 03:09:11.457162 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":418,"threads_started":1}
I20260807 03:09:11.461817 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:09:11.466102 23177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802427dbce0f4815af411c22e72ec80f" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35589 } }
I20260807 03:09:11.466513 23177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:11.466620 23177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802427dbce0f4815af411c22e72ec80f, State: Initialized, Role: FOLLOWER
I20260807 03:09:11.467157 23177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802427dbce0f4815af411c22e72ec80f" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35589 } }
I20260807 03:09:11.467384 23177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:09:11.467530 23177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:09:11.467731 23177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:11.469899 23177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802427dbce0f4815af411c22e72ec80f" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35589 } }
I20260807 03:09:11.470465 23177 leader_election.cc:302] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 802427dbce0f4815af411c22e72ec80f; no voters:
I20260807 03:09:11.471060 23177 leader_election.cc:288] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:09:11.471216 23182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:11.471966 23182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [term 1 LEADER]: Becoming Leader. State: Replica: 802427dbce0f4815af411c22e72ec80f, State: Running, Role: LEADER
I20260807 03:09:11.472489 23182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802427dbce0f4815af411c22e72ec80f" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35589 } }
I20260807 03:09:11.473064 23177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:09:11.475448 23184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 802427dbce0f4815af411c22e72ec80f. Latest consensus state: current_term: 1 leader_uuid: "802427dbce0f4815af411c22e72ec80f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802427dbce0f4815af411c22e72ec80f" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35589 } } }
I20260807 03:09:11.475395 23183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "802427dbce0f4815af411c22e72ec80f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802427dbce0f4815af411c22e72ec80f" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35589 } } }
I20260807 03:09:11.475858 23184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:11.475868 23183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:11.481127 23188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:09:11.486192 23188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:09:11.495018 23188 catalog_manager.cc:1347] Generated new cluster ID: 298b1994f4bf4a4b8d9f29752e365367
I20260807 03:09:11.495141 23188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:09:11.513808 23188 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:09:11.515048 23188 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:09:11.528954 23188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 802427dbce0f4815af411c22e72ec80f: Generated new TSK 0
I20260807 03:09:11.529747 23188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:09:11.551460 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:11.543322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57940 (local address 127.17.31.62:35589)
0807 03:09:11.543552 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:11.543562 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:11.544061 (+ 499us) server_negotiation.cc:407] Connection header received
0807 03:09:11.544285 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:11.544288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:11.544358 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:11.544481 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0807 03:09:11.545033 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:11.545758 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:11.547800 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:11.548214 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:11.548301 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:11.548873 (+ 572us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 03:09:11.550811 (+ 1938us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35567/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 03:09:11.551149 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57940: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35567/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260807 03:09:11.551709 23207 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:11.543299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35589 (local address 127.0.0.1:57940)
0807 03:09:11.543952 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:11.543972 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:11.544138 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:11.544527 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:11.544532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:11.544545 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0807 03:09:11.544871 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:11.544883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:11.545898 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:11.545904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:11.547664 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:11.547680 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:11.548402 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:11.548409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:11.548602 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:11.548650 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 03:09:11.551022 (+ 2372us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35567/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 03:09:11.551257 (+ 235us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.31.62:35589: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35567/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":351,"threads_started":1}
W20260807 03:09:11.552049 23180 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57940: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35567/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260807 03:09:11.552531 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 23107
I20260807 03:09:11.553654 23172 generic_service.cc:196] Checking for leaks (request via RPC)
E20260807 03:09:11.563771 23208 webserver.cc:533] Webserver: sslize failed:
E20260807 03:09:11.563860 23208 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 03:09:11.644676 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 23107
2026-08-07T03:09:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (711 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-07T03:09:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:09:11Z Disabled control of system clock
I20260807 03:09:11.729595 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:45207
--webserver_interface=127.17.31.62
--webserver_port=0
--builtin_ntp_servers=127.17.31.20:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:45207
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/testchainca.cert with env {}
W20260807 03:09:12.056144 23213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:12.056427 23213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:12.056512 23213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:12.063567 23213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:09:12.063665 23213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:12.063702 23213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:12.063751 23213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:09:12.063799 23213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 03:09:12.063853 23213 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 03:09:12.063892 23213 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260807 03:09:12.073474 23213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:45207
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:45207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:12.074740 23213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:12.075968 23213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:12.088001 23221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:12.087978 23219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:12.088181 23218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:12.088954 23213 server_base.cc:1034] running on GCE node
I20260807 03:09:12.089712 23213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:12.091141 23213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:12.092381 23213 hybrid_clock.cc:648] HybridClock initialized: now 1786072152092337 us; error 48 us; skew 500 ppm
I20260807 03:09:12.092723 23213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:09:12.094937 23213 webserver.cc:460] Webserver started at http://127.17.31.62:41935/ using document root <none> and password file <none>
I20260807 03:09:12.095643 23213 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:12.095784 23213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:12.096094 23213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:12.098966 23213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "eba13ea6f13c4de79267e8bde8453c95"
format_stamp: "Formatted at 2026-08-07 03:09:12 on dist-test-slave-tm1g"
server_key: "51f8ab7c666adf032d9aeb30e124d11c"
server_key_iv: "ce37ccdf79e0c3f5b2560822cede2332"
server_key_version: "encryptionkey@0"
I20260807 03:09:12.099761 23213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "eba13ea6f13c4de79267e8bde8453c95"
format_stamp: "Formatted at 2026-08-07 03:09:12 on dist-test-slave-tm1g"
server_key: "51f8ab7c666adf032d9aeb30e124d11c"
server_key_iv: "ce37ccdf79e0c3f5b2560822cede2332"
server_key_version: "encryptionkey@0"
I20260807 03:09:12.104800 23213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 03:09:12.108070 23227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:12.109308 23213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 03:09:12.109509 23213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "eba13ea6f13c4de79267e8bde8453c95"
format_stamp: "Formatted at 2026-08-07 03:09:12 on dist-test-slave-tm1g"
server_key: "51f8ab7c666adf032d9aeb30e124d11c"
server_key_iv: "ce37ccdf79e0c3f5b2560822cede2332"
server_key_version: "encryptionkey@0"
I20260807 03:09:12.109728 23213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:12.142218 23213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:12.143095 23213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:12.143392 23213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:12.160982 23213 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:45207
I20260807 03:09:12.161015 23281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:45207 every 8 connection(s)
I20260807 03:09:12.162238 23213 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:09:12.165856 23282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:12.167701 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 23213
I20260807 03:09:12.167928 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:09:12.168578 17532 external_mini_cluster.cc:1428] Setting key 7bd281564c40f52907b0c11acb0efb36
I20260807 03:09:12.177165 23282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95: Bootstrap starting.
I20260807 03:09:12.180771 23282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:12.181795 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:12.170502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.17.31.62:45207)
0807 03:09:12.171517 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:12.171527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:12.171568 (+ 41us) server_negotiation.cc:407] Connection header received
0807 03:09:12.173118 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:12.173138 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:12.173568 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:12.173900 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:12.174532 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:12.175796 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:12.177573 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:12.178218 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:12.178425 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:12.179622 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:12.179629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:12.179641 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:12.180147 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:12.180231 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:12.180251 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:12.180339 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:12.180591 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 03:09:12.180889 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":402,"threads_started":1}
I20260807 03:09:12.181843 23282 log.cc:824] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:12.185051 23282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95: No bootstrap required, opened a new log
I20260807 03:09:12.185947 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:09:12.193570 23282 raft_consensus.cc:348] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba13ea6f13c4de79267e8bde8453c95" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45207 } }
I20260807 03:09:12.193873 23282 raft_consensus.cc:374] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:12.193946 23282 raft_consensus.cc:729] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba13ea6f13c4de79267e8bde8453c95, State: Initialized, Role: FOLLOWER
I20260807 03:09:12.194455 23282 consensus_queue.cc:260] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba13ea6f13c4de79267e8bde8453c95" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45207 } }
I20260807 03:09:12.194654 23282 raft_consensus.cc:388] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:09:12.194748 23282 raft_consensus.cc:482] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:09:12.194869 23282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:12.196878 23282 raft_consensus.cc:504] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba13ea6f13c4de79267e8bde8453c95" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45207 } }
I20260807 03:09:12.197307 23282 leader_election.cc:302] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eba13ea6f13c4de79267e8bde8453c95; no voters:
I20260807 03:09:12.198004 23282 leader_election.cc:288] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:09:12.198101 23287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:12.198877 23287 raft_consensus.cc:686] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [term 1 LEADER]: Becoming Leader. State: Replica: eba13ea6f13c4de79267e8bde8453c95, State: Running, Role: LEADER
I20260807 03:09:12.199434 23287 consensus_queue.cc:237] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba13ea6f13c4de79267e8bde8453c95" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45207 } }
I20260807 03:09:12.200196 23282 sys_catalog.cc:564] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:09:12.202062 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eba13ea6f13c4de79267e8bde8453c95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba13ea6f13c4de79267e8bde8453c95" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45207 } } }
I20260807 03:09:12.202543 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:12.202538 23288 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba13ea6f13c4de79267e8bde8453c95. Latest consensus state: current_term: 1 leader_uuid: "eba13ea6f13c4de79267e8bde8453c95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba13ea6f13c4de79267e8bde8453c95" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45207 } } }
I20260807 03:09:12.202864 23288 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95 [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:12.207597 23295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:09:12.212798 23295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:09:12.221227 23295 catalog_manager.cc:1347] Generated new cluster ID: ac8bce41aa8549049867aaea9911e64e
I20260807 03:09:12.221342 23295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:09:12.249470 23295 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:09:12.250823 23295 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:09:12.258581 23295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eba13ea6f13c4de79267e8bde8453c95: Generated new TSK 0
I20260807 03:09:12.259528 23295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:09:12.351125 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:12.280919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39944 (local address 127.17.31.62:45207)
0807 03:09:12.281163 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:12.281173 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:12.281594 (+ 421us) server_negotiation.cc:407] Connection header received
0807 03:09:12.281868 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:12.281873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:12.281961 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:12.282116 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0807 03:09:12.282632 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:12.283797 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:12.285125 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:12.285849 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:12.285977 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:12.286435 (+ 458us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 03:09:12.350338 (+ 63903us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 03:09:12.350782 (+ 444us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39944: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":66}
W20260807 03:09:12.351246 23312 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:12.280820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45207 (local address 127.0.0.1:39944)
0807 03:09:12.281475 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:12.281497 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:12.281649 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:12.282149 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:12.282153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:12.282163 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0807 03:09:12.282483 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:12.282494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:12.283945 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:12.283949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:12.284990 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:12.285000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:12.286016 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:12.286020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:12.286137 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:12.286168 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 03:09:12.350641 (+ 64473us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 03:09:12.350885 (+ 244us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.31.62:45207: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":325,"threads_started":1}
W20260807 03:09:12.351279 23285 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39944: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260807 03:09:12.352628 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 23213
I20260807 03:09:12.353956 23277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:12.454783 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 23213
2026-08-07T03:09:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (790 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (407 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (406 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:09:13 dist-test-slave-tm1g krb5kdc[23327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:09:13 dist-test-slave-tm1g krb5kdc[23327](info): set up 2 sockets
Aug 07 03:09:13 dist-test-slave-tm1g krb5kdc[23327](info): commencing operation
krb5kdc: starting...
W20260807 03:09:14.384256 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:09:14 dist-test-slave-tm1g krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:09:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:09:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:09:14.587772 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:42223
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:42223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:14.906129 23341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:14.906412 23341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:14.906500 23341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:14.913311 23341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:09:14.913403 23341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:14.913447 23341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:14.913494 23341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:09:14.913530 23341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:09:14.922719 23341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:42223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:42223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:14.923825 23341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:14.924985 23341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:14.936306 23349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:14.936420 23347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:14.936394 23346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:14.937258 23341 server_base.cc:1034] running on GCE node
I20260807 03:09:14.937975 23341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:14.939277 23341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:14.940471 23341 hybrid_clock.cc:648] HybridClock initialized: now 1786072154940411 us; error 58 us; skew 500 ppm
I20260807 03:09:14.940798 23341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:14 dist-test-slave-tm1g krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072154, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:14.943773 23341 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:09:14.945147 23341 webserver.cc:460] Webserver started at http://127.17.31.62:42511/ using document root <none> and password file <none>
I20260807 03:09:14.945768 23341 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:14.945865 23341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:14.946111 23341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:14.949193 23341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "be7a3cac301048ee9b62c6ad6310bd33"
format_stamp: "Formatted at 2026-08-07 03:09:14 on dist-test-slave-tm1g"
server_key: "73af4c1161ec1e23d06848d9d0446fb8"
server_key_iv: "d5d96186cbe67c0bb234c8e198f58e4e"
server_key_version: "encryptionkey@0"
I20260807 03:09:14.949911 23341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "be7a3cac301048ee9b62c6ad6310bd33"
format_stamp: "Formatted at 2026-08-07 03:09:14 on dist-test-slave-tm1g"
server_key: "73af4c1161ec1e23d06848d9d0446fb8"
server_key_iv: "d5d96186cbe67c0bb234c8e198f58e4e"
server_key_version: "encryptionkey@0"
I20260807 03:09:14.954788 23341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 03:09:14.957978 23356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:14.959286 23341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:09:14.959476 23341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "be7a3cac301048ee9b62c6ad6310bd33"
format_stamp: "Formatted at 2026-08-07 03:09:14 on dist-test-slave-tm1g"
server_key: "73af4c1161ec1e23d06848d9d0446fb8"
server_key_iv: "d5d96186cbe67c0bb234c8e198f58e4e"
server_key_version: "encryptionkey@0"
I20260807 03:09:14.959667 23341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:14.986671 23341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:14.990471 23341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:14.990751 23341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:15.008644 23341 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:42223
I20260807 03:09:15.008687 23412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:42223 every 8 connection(s)
I20260807 03:09:15.009905 23341 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:09:15.013378 23413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:15.015836 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 23341
I20260807 03:09:15.016080 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:09:15.016705 17532 external_mini_cluster.cc:1428] Setting key 5985663b4bc63409fa4262f3fa6e4592
I20260807 03:09:15.026394 23413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33: Bootstrap starting.
I20260807 03:09:15.029320 23413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:15.030383 23413 log.cc:824] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33: Log is configured to *not* fsync() on all Append() calls
Aug 07 03:09:15 dist-test-slave-tm1g krb5kdc[23327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:15.033257 23413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33: No bootstrap required, opened a new log
I20260807 03:09:15.040726 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:15.018705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.17.31.62:42223)
0807 03:09:15.019479 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:15.019494 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:15.019541 (+ 47us) server_negotiation.cc:407] Connection header received
0807 03:09:15.021789 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:15.021816 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:15.022296 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:15.022748 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:15.024822 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:15.026726 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:15.027981 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:15.028639 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:15.028839 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:15.032776 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:15.032788 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:15.032805 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:15.032850 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:15.036840 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:15.037460 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:15.037469 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:15.037476 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:15.037678 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:15.038149 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:15.038153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:15.038155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:15.038714 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:15.038933 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:15.039376 (+ 443us) server_negotiation.cc:299] Negotiation successful
0807 03:09:15.039679 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":332,"threads_started":1}
I20260807 03:09:15.042817 23413 raft_consensus.cc:348] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7a3cac301048ee9b62c6ad6310bd33" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42223 } }
I20260807 03:09:15.043167 23413 raft_consensus.cc:374] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:15.043313 23413 raft_consensus.cc:729] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be7a3cac301048ee9b62c6ad6310bd33, State: Initialized, Role: FOLLOWER
I20260807 03:09:15.043874 23413 consensus_queue.cc:260] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7a3cac301048ee9b62c6ad6310bd33" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42223 } }
I20260807 03:09:15.044078 23413 raft_consensus.cc:388] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:09:15.044170 23413 raft_consensus.cc:482] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:09:15.044288 23413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:15.046578 23413 raft_consensus.cc:504] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7a3cac301048ee9b62c6ad6310bd33" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42223 } }
I20260807 03:09:15.047067 23413 leader_election.cc:302] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be7a3cac301048ee9b62c6ad6310bd33; no voters:
I20260807 03:09:15.047530 23413 leader_election.cc:288] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:09:15.047676 23418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:15.048341 23418 raft_consensus.cc:686] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [term 1 LEADER]: Becoming Leader. State: Replica: be7a3cac301048ee9b62c6ad6310bd33, State: Running, Role: LEADER
I20260807 03:09:15.048942 23418 consensus_queue.cc:237] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7a3cac301048ee9b62c6ad6310bd33" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42223 } }
I20260807 03:09:15.049489 23413 sys_catalog.cc:564] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:09:15.051934 23420 sys_catalog.cc:455] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be7a3cac301048ee9b62c6ad6310bd33. Latest consensus state: current_term: 1 leader_uuid: "be7a3cac301048ee9b62c6ad6310bd33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7a3cac301048ee9b62c6ad6310bd33" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42223 } } }
I20260807 03:09:15.052340 23420 sys_catalog.cc:458] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:15.051888 23419 sys_catalog.cc:455] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be7a3cac301048ee9b62c6ad6310bd33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7a3cac301048ee9b62c6ad6310bd33" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42223 } } }
I20260807 03:09:15.052918 23419 sys_catalog.cc:458] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33 [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:15.054112 23427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:09:15.059386 23427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:09:15.067312 23427 catalog_manager.cc:1347] Generated new cluster ID: 912134f6258545fcb0cc1ca7f685dfab
I20260807 03:09:15.067452 23427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:09:15.075479 23427 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:09:15.076682 23427 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:09:15.097470 23427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be7a3cac301048ee9b62c6ad6310bd33: Generated new TSK 0
I20260807 03:09:15.098371 23427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:09:15.152917 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42223
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:15.477012 23441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:15.477274 23441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:15.477419 23441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:15.484560 23441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:15.484653 23441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:15.484851 23441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:09:15.494488 23441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:15.495703 23441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:15.497010 23441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:15.510069 23446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:15.509999 23447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:15.510144 23449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:15.511130 23441 server_base.cc:1034] running on GCE node
I20260807 03:09:15.511699 23441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:15.512548 23441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:15.513767 23441 hybrid_clock.cc:648] HybridClock initialized: now 1786072155513722 us; error 60 us; skew 500 ppm
I20260807 03:09:15.514062 23441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:15 dist-test-slave-tm1g krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072155, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:15.517001 23441 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:09:15.518340 23441 webserver.cc:460] Webserver started at http://127.17.31.1:36311/ using document root <none> and password file <none>
I20260807 03:09:15.519003 23441 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:15.519147 23441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:15.519454 23441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:15.522192 23441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "e3ebf99d85f54c58b25474406069aa6f"
format_stamp: "Formatted at 2026-08-07 03:09:15 on dist-test-slave-tm1g"
server_key: "15e5452f651c4c0f58c75a84bb8063d1"
server_key_iv: "141cecf3d6e44c830f5b5c2fa6fbb885"
server_key_version: "encryptionkey@0"
I20260807 03:09:15.523005 23441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "e3ebf99d85f54c58b25474406069aa6f"
format_stamp: "Formatted at 2026-08-07 03:09:15 on dist-test-slave-tm1g"
server_key: "15e5452f651c4c0f58c75a84bb8063d1"
server_key_iv: "141cecf3d6e44c830f5b5c2fa6fbb885"
server_key_version: "encryptionkey@0"
I20260807 03:09:15.527756 23441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 03:09:15.530853 23456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:15.532189 23441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:09:15.532346 23441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "e3ebf99d85f54c58b25474406069aa6f"
format_stamp: "Formatted at 2026-08-07 03:09:15 on dist-test-slave-tm1g"
server_key: "15e5452f651c4c0f58c75a84bb8063d1"
server_key_iv: "141cecf3d6e44c830f5b5c2fa6fbb885"
server_key_version: "encryptionkey@0"
I20260807 03:09:15.532554 23441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:15.547497 23441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:15.551278 23441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:15.551554 23441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:15.552428 23441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:15.553793 23441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:15.553893 23441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:15.554013 23441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:15.554059 23441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:15.586102 23441 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:40891
I20260807 03:09:15.586145 23574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:40891 every 8 connection(s)
I20260807 03:09:15.587399 23441 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:09:15.592541 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 23441
I20260807 03:09:15.592765 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:09:15.593573 17532 external_mini_cluster.cc:1428] Setting key 3fcf6f054f36662572ed70ae91aa49fb
Aug 07 03:09:15 dist-test-slave-tm1g krb5kdc[23327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072155, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:15.612820 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:15.589894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:43583 (local address 127.17.31.62:42223)
0807 03:09:15.590379 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:15.590383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:15.591315 (+ 932us) server_negotiation.cc:407] Connection header received
0807 03:09:15.593100 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:15.593107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:15.593193 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:15.593323 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:15.594898 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:15.595936 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:15.598516 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:15.598910 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:15.599005 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:15.605827 (+ 6822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:15.605836 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:15.605844 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:15.605888 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:15.609401 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:15.610177 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:15.610185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:15.610191 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:15.610380 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:15.610794 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:15.610801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:15.610807 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:15.611217 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:15.611366 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:15.612031 (+ 665us) server_negotiation.cc:299] Negotiation successful
0807 03:09:15.612287 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":320,"thread_start_us":217,"threads_started":1}
I20260807 03:09:15.613431 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:15.590469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:42223 (local address 127.17.31.1:43583)
0807 03:09:15.591160 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:15.591201 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:15.592803 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:15.593499 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:15.593507 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:15.594058 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:15.594740 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:15.594759 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:15.596100 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:15.596107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:15.598377 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:15.598394 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:15.599133 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:15.599143 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:15.599468 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:15.603460 (+ 3992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:15.603482 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:15.605584 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:15.609618 (+ 4034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:15.609624 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:15.609636 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:15.610037 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:15.610486 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:15.610489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:15.610491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:15.610674 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:15.611333 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:15.611338 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:15.611657 (+ 319us) client_negotiation.cc:769] Sending connection context
0807 03:09:15.611885 (+ 228us) client_negotiation.cc:240] Negotiation successful
0807 03:09:15.612260 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":284,"threads_started":1}
I20260807 03:09:15.615079 23575 heartbeater.cc:347] Connected to a master server at 127.17.31.62:42223
I20260807 03:09:15.615427 23575 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:15.616325 23575 heartbeater.cc:511] Master 127.17.31.62:42223 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
I20260807 03:09:15.618639 23373 ts_manager.cc:194] Registered new tserver with Master: e3ebf99d85f54c58b25474406069aa6f (127.17.31.1:40891)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
I20260807 03:09:15.620187 23373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:43583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:09:15.643486 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42223
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:15.979984 23581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:15.980252 23581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:15.980347 23581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:15.987241 23581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:15.987342 23581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:15.987458 23581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:09:15.996600 23581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:15.997772 23581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:15.999014 23581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:16.011530 23588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:16.011509 23587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:16.012249 23590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:16.014438 23581 server_base.cc:1034] running on GCE node
I20260807 03:09:16.015089 23581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:16.016110 23581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:16.017333 23581 hybrid_clock.cc:648] HybridClock initialized: now 1786072156017291 us; error 36 us; skew 500 ppm
I20260807 03:09:16.017661 23581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:16 dist-test-slave-tm1g krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072156, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:16.021246 23581 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:09:16.026356 23581 webserver.cc:460] Webserver started at http://127.17.31.2:46333/ using document root <none> and password file <none>
I20260807 03:09:16.027279 23581 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:16.027469 23581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:16.027874 23581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:16.031154 23581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "47d3ecc1d4b54a22a344cfdf4a50a098"
format_stamp: "Formatted at 2026-08-07 03:09:16 on dist-test-slave-tm1g"
server_key: "01e6ee15344c3199343829fde71a3e1e"
server_key_iv: "8916cb32ebaee2f20ea35397a1a3ba07"
server_key_version: "encryptionkey@0"
I20260807 03:09:16.031919 23581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "47d3ecc1d4b54a22a344cfdf4a50a098"
format_stamp: "Formatted at 2026-08-07 03:09:16 on dist-test-slave-tm1g"
server_key: "01e6ee15344c3199343829fde71a3e1e"
server_key_iv: "8916cb32ebaee2f20ea35397a1a3ba07"
server_key_version: "encryptionkey@0"
I20260807 03:09:16.036952 23581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 03:09:16.040337 23597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:16.041530 23581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:09:16.041702 23581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "47d3ecc1d4b54a22a344cfdf4a50a098"
format_stamp: "Formatted at 2026-08-07 03:09:16 on dist-test-slave-tm1g"
server_key: "01e6ee15344c3199343829fde71a3e1e"
server_key_iv: "8916cb32ebaee2f20ea35397a1a3ba07"
server_key_version: "encryptionkey@0"
I20260807 03:09:16.041896 23581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:16.056705 23581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:16.060883 23581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:16.061208 23581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:16.062103 23581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:16.063431 23581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:16.063529 23581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:16.063678 23581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:16.063756 23581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:16.096050 23581 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:38347
I20260807 03:09:16.096086 23714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:38347 every 8 connection(s)
I20260807 03:09:16.097352 23581 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:09:16.102614 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 23581
I20260807 03:09:16.102823 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:09:16.103550 17532 external_mini_cluster.cc:1428] Setting key 2bccc43f1e661bb31e1203d7cd301434
Aug 07 03:09:16 dist-test-slave-tm1g krb5kdc[23327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072156, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:16.118610 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:16.100017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:51151 (local address 127.17.31.62:42223)
0807 03:09:16.100196 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:16.100201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:16.101338 (+ 1137us) server_negotiation.cc:407] Connection header received
0807 03:09:16.103415 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:16.103421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:16.103517 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:16.103630 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:16.105222 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.105971 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.107551 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.107962 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.108096 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:16.111912 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:16.111917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:16.111920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:16.111952 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:16.115413 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:16.116099 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:16.116107 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:16.116113 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:16.116286 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:16.116674 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:16.116682 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:16.116688 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:16.117050 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:16.117200 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:16.117856 (+ 656us) server_negotiation.cc:299] Negotiation successful
0807 03:09:16.118130 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 03:09:16.119189 23717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:16.100474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:42223 (local address 127.17.31.2:51151)
0807 03:09:16.101181 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:16.101219 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:16.103048 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:16.103856 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:16.103864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:16.104366 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:16.105061 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:16.105080 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.106121 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.106127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:16.107395 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:16.107407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.108180 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.108189 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:16.108415 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:16.109084 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:16.109107 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:16.111700 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:16.115547 (+ 3847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:16.115553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:16.115564 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:16.115966 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:16.116395 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:16.116398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:16.116400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:16.116562 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:16.117164 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:16.117169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:16.117467 (+ 298us) client_negotiation.cc:769] Sending connection context
0807 03:09:16.117682 (+ 215us) client_negotiation.cc:240] Negotiation successful
0807 03:09:16.118056 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":295,"threads_started":1}
I20260807 03:09:16.120652 23715 heartbeater.cc:347] Connected to a master server at 127.17.31.62:42223
I20260807 03:09:16.120987 23715 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:16.121810 23715 heartbeater.cc:511] Master 127.17.31.62:42223 requested a full tablet report, sending...
I20260807 03:09:16.123435 23373 ts_manager.cc:194] Registered new tserver with Master: 47d3ecc1d4b54a22a344cfdf4a50a098 (127.17.31.2:38347)
I20260807 03:09:16.124243 23373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:51151
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:09:16.153115 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42223
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:16.472999 23720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:16.473310 23720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:16.473495 23720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:16.480639 23720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:16.480786 23720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:16.480952 23720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:09:16.491207 23720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:16.492478 23720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:16.493791 23720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:16.506807 23725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:16.506728 23726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:16.506776 23728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:16.507776 23720 server_base.cc:1034] running on GCE node
I20260807 03:09:16.508342 23720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:16.509208 23720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:16.510402 23720 hybrid_clock.cc:648] HybridClock initialized: now 1786072156510363 us; error 60 us; skew 500 ppm
I20260807 03:09:16.510713 23720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:16 dist-test-slave-tm1g krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072156, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:16.513644 23720 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:09:16.515129 23720 webserver.cc:460] Webserver started at http://127.17.31.3:36805/ using document root <none> and password file <none>
I20260807 03:09:16.515800 23720 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:16.515946 23720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:16.516274 23720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:16.519127 23720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "8c49ac806f5846de85d564b938980386"
format_stamp: "Formatted at 2026-08-07 03:09:16 on dist-test-slave-tm1g"
server_key: "bf4e8a312b3b4a93e3bdf22f8dce967f"
server_key_iv: "4bb0022809a357f98ca90f6745569652"
server_key_version: "encryptionkey@0"
I20260807 03:09:16.519879 23720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "8c49ac806f5846de85d564b938980386"
format_stamp: "Formatted at 2026-08-07 03:09:16 on dist-test-slave-tm1g"
server_key: "bf4e8a312b3b4a93e3bdf22f8dce967f"
server_key_iv: "4bb0022809a357f98ca90f6745569652"
server_key_version: "encryptionkey@0"
I20260807 03:09:16.524751 23720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 03:09:16.528015 23735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:16.529191 23720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 03:09:16.529358 23720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "8c49ac806f5846de85d564b938980386"
format_stamp: "Formatted at 2026-08-07 03:09:16 on dist-test-slave-tm1g"
server_key: "bf4e8a312b3b4a93e3bdf22f8dce967f"
server_key_iv: "4bb0022809a357f98ca90f6745569652"
server_key_version: "encryptionkey@0"
I20260807 03:09:16.529557 23720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:16.556977 23720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:16.560832 23720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:16.561120 23720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:16.562033 23720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:16.563356 23720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:16.563477 23720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:16.563629 23720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:16.563700 23720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:16.596513 23720 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:46491
I20260807 03:09:16.596596 23856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:46491 every 8 connection(s)
I20260807 03:09:16.597731 23720 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:09:16.602998 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 23720
I20260807 03:09:16.603220 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:09:16.603797 17532 external_mini_cluster.cc:1428] Setting key 9564a01b011160b9c997d805a7e4bc55
Aug 07 03:09:16 dist-test-slave-tm1g krb5kdc[23327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072156, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:16.617056 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:16.600185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:53899 (local address 127.17.31.62:42223)
0807 03:09:16.600394 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:16.600400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:16.601570 (+ 1170us) server_negotiation.cc:407] Connection header received
0807 03:09:16.603702 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:16.603706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:16.603783 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:16.603897 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:16.605887 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.606902 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.608415 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.608798 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.608887 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:16.612165 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:16.612170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:16.612173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:16.612204 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:16.614602 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:16.615254 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:16.615258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:16.615260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:16.615380 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:16.615722 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:16.615725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:16.615727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:16.615971 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:16.616085 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:16.616657 (+ 572us) server_negotiation.cc:299] Negotiation successful
0807 03:09:16.616803 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 03:09:16.618181 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:16.600681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:42223 (local address 127.17.31.3:53899)
0807 03:09:16.601411 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:16.601454 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:16.603402 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:16.604157 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:16.604170 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:16.604879 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:16.605691 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:16.605712 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.607069 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.607075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:16.608296 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:16.608308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.608976 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.608981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:16.609171 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:16.609854 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:16.609876 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:16.611967 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:16.614720 (+ 2753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:16.614726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:16.614737 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:16.615158 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:16.615470 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:16.615473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:16.615475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:16.615635 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:16.616084 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:16.616090 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:16.616369 (+ 279us) client_negotiation.cc:769] Sending connection context
0807 03:09:16.616590 (+ 221us) client_negotiation.cc:240] Negotiation successful
0807 03:09:16.616936 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":323,"threads_started":1}
I20260807 03:09:16.619658 23857 heartbeater.cc:347] Connected to a master server at 127.17.31.62:42223
I20260807 03:09:16.619982 23857 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:16.620605 23857 heartbeater.cc:511] Master 127.17.31.62:42223 requested a full tablet report, sending...
I20260807 03:09:16.621917 23372 ts_manager.cc:194] Registered new tserver with Master: 8c49ac806f5846de85d564b938980386 (127.17.31.3:46491)
I20260807 03:09:16.622802 23372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:53899
I20260807 03:09:16.623157 23575 heartbeater.cc:503] Master 127.17.31.62:42223 was elected leader, sending a full tablet report...
I20260807 03:09:16.631790 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/test-admin.keytab.
Aug 07 03:09:16 dist-test-slave-tm1g krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:16.659993 17532 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260807 03:09:16.665565 23865 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:16.662836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33906 (local address 127.17.31.1:40891)
0807 03:09:16.663582 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:16.663589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:16.663616 (+ 27us) server_negotiation.cc:407] Connection header received
0807 03:09:16.663699 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:16.663706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:16.663938 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:16.664084 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:16.665130 (+ 1046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33906: BlockingRecv error: recv got EOF from 127.0.0.1:33906 (error 108)
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":368,"threads_started":1}
W20260807 03:09:16.665568 17532 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/krb5cc' of type FILE
Aug 07 03:09:16 dist-test-slave-tm1g krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:16.667433 17532 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 03:09:16 dist-test-slave-tm1g krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:16.668784 17532 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 03:09:16 dist-test-slave-tm1g krb5kdc[23327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:09:16.684324 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:16.670510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33910 (local address 127.17.31.1:40891)
0807 03:09:16.670713 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:16.670718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:16.671118 (+ 400us) server_negotiation.cc:407] Connection header received
0807 03:09:16.671248 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:16.671252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:16.671311 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:16.671391 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:16.672336 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.673293 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.674645 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.675216 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.675313 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:16.679185 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:16.679196 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:16.679211 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:16.679244 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:16.681829 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:16.682500 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:16.682508 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:16.682512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:16.682629 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:16.682957 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:16.682959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:16.682961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:16.683314 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:16.683530 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:16.683874 (+ 344us) server_negotiation.cc:299] Negotiation successful
0807 03:09:16.684075 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260807 03:09:16.690305 23865 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:16.688478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.17.31.1:40891)
0807 03:09:16.688696 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:16.688704 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:16.688991 (+ 287us) server_negotiation.cc:407] Connection header received
0807 03:09:16.689181 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:16.689188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:16.689257 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:16.689355 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:16.690027 (+ 672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33920: BlockingRecv error: recv got EOF from 127.0.0.1:33920 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260807 03:09:16.690531 17532 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/krb5cc' of type FILE
Aug 07 03:09:16 dist-test-slave-tm1g krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:16.691994 17532 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 03:09:16 dist-test-slave-tm1g krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:16.693279 17532 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 03:09:16 dist-test-slave-tm1g krb5kdc[23327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:09:16.706820 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:16.694531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33922 (local address 127.17.31.1:40891)
0807 03:09:16.694731 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:16.694738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:16.694981 (+ 243us) server_negotiation.cc:407] Connection header received
0807 03:09:16.695094 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:16.695098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:16.695150 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:16.695224 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:16.696081 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.696752 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.697862 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.698260 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.698448 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:16.701781 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:16.701785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:16.701789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:16.701831 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:16.704501 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:16.705083 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:16.705087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:16.705089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:16.705202 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:16.705547 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:16.705552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:16.705554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:16.705801 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:16.705998 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:16.706258 (+ 260us) server_negotiation.cc:299] Negotiation successful
0807 03:09:16.706487 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260807 03:09:16.710974 23865 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:16.709208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33926 (local address 127.17.31.1:40891)
0807 03:09:16.709403 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:16.709411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:16.709679 (+ 268us) server_negotiation.cc:407] Connection header received
0807 03:09:16.709851 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:16.709857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:16.709929 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:16.710026 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:16.710759 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33926: BlockingRecv error: recv got EOF from 127.0.0.1:33926 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260807 03:09:16.711239 17532 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786072106928531-17532-0/krb5kdc/krb5cc' of type FILE
Aug 07 03:09:16 dist-test-slave-tm1g krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:16.712656 17532 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 03:09:16 dist-test-slave-tm1g krb5kdc[23327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:16.713898 17532 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 03:09:16 dist-test-slave-tm1g krb5kdc[23327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:09:16.727456 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:16.715306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33928 (local address 127.17.31.1:40891)
0807 03:09:16.715489 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:16.715495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:16.715684 (+ 189us) server_negotiation.cc:407] Connection header received
0807 03:09:16.715790 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:16.715793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:16.715846 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:16.715922 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:16.716856 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.717557 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.718840 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.719250 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.719344 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:16.722740 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:16.722745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:16.722749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:16.722778 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:16.725172 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:16.725764 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:16.725767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:16.725769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:16.725906 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:16.726210 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:16.726213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:16.726215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:16.726529 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:16.726749 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:16.726957 (+ 208us) server_negotiation.cc:299] Negotiation successful
0807 03:09:16.727144 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 03:09:16.729022 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 23441
I20260807 03:09:16.740648 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:16.729694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.17.31.1:40891)
0807 03:09:16.729832 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:16.729836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:16.730179 (+ 343us) server_negotiation.cc:407] Connection header received
0807 03:09:16.730290 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:16.730293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:16.730392 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:16.730464 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:16.731298 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.731941 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.733194 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.733668 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.733757 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:16.735778 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:16.735783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:16.735786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:16.735848 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:16.738204 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:16.738843 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:16.738847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:16.738849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:16.738973 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:16.739328 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:16.739331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:16.739333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:16.739582 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:16.739842 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:16.740138 (+ 296us) server_negotiation.cc:299] Negotiation successful
0807 03:09:16.740282 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 03:09:16.741266 23570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:16.824057 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 23441
I20260807 03:09:16.842792 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 23581
Aug 07 03:09:16 dist-test-slave-tm1g krb5kdc[23327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:09:16.857771 23878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:16.843774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42930 (local address 127.17.31.2:38347)
0807 03:09:16.844308 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:16.844315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:16.844353 (+ 38us) server_negotiation.cc:407] Connection header received
0807 03:09:16.844414 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:16.844420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:16.844659 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:16.844795 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:16.845899 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.846979 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.848144 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.848678 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.848772 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:16.852652 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:16.852661 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:16.852674 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:16.852705 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:16.855368 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:16.855989 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:16.855997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:16.855999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:16.856126 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:16.856472 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:16.856475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:16.856477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:16.856789 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:16.857017 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:16.857269 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 03:09:16.857477 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":293,"threads_started":1}
I20260807 03:09:16.859062 23710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:16.941516 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 23581
I20260807 03:09:16.958633 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 23720
Aug 07 03:09:16 dist-test-slave-tm1g krb5kdc[23327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:09:16.974467 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:16.959624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.17.31.3:46491)
0807 03:09:16.960293 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:16.960300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:16.960328 (+ 28us) server_negotiation.cc:407] Connection header received
0807 03:09:16.960389 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:16.960395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:16.960609 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:16.960775 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:16.961917 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.963027 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.964263 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:16.964837 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:16.964973 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:16.969061 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:16.969071 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:16.969086 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:16.969119 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:16.971736 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:16.972409 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:16.972424 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:16.972427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:16.972544 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:16.972912 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:16.972918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:16.972921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:16.973252 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:16.973565 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:16.973861 (+ 296us) server_negotiation.cc:299] Negotiation successful
0807 03:09:16.974113 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":345,"threads_started":1}
I20260807 03:09:16.975719 23851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:17.061137 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 23720
I20260807 03:09:17.078599 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 23341
I20260807 03:09:17.079771 23406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:17.184033 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 23341
2026-08-07T03:09:17Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3919 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:09:17 dist-test-slave-tm1g krb5kdc[23885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:09:17 dist-test-slave-tm1g krb5kdc[23885](info): set up 2 sockets
Aug 07 03:09:17 dist-test-slave-tm1g krb5kdc[23885](info): commencing operation
krb5kdc: starting...
W20260807 03:09:18.405838 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:09:18 dist-test-slave-tm1g krb5kdc[23885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:09:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:09:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.62.keytab.
Entry for principal oryx/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.62.keytab.
I20260807 03:09:18.783097 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:44907
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.62.keytab
--principal=oryx/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:44907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:19.337198 23898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:19.337627 23898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:19.337759 23898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:19.349169 23898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:09:19.349532 23898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:19.349694 23898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:19.349819 23898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:09:19.349907 23898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:09:19.365340 23898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:44907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.62.keytab
--principal=oryx/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:44907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:19.367439 23898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:19.369647 23898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:19.388032 23903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:19.387899 23906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:19.387969 23904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:19.389622 23898 server_base.cc:1034] running on GCE node
I20260807 03:09:19.390908 23898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:19.393139 23898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:19.394425 23898 hybrid_clock.cc:648] HybridClock initialized: now 1786072159394338 us; error 83 us; skew 500 ppm
I20260807 03:09:19.394897 23898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:19 dist-test-slave-tm1g krb5kdc[23885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072159, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:19.399989 23898 init.cc:377] Logged in from keytab as oryx/127.17.31.62@KRBTEST.COM (short username oryx)
I20260807 03:09:19.402201 23898 webserver.cc:460] Webserver started at http://127.17.31.62:34093/ using document root <none> and password file <none>
I20260807 03:09:19.403362 23898 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:19.403524 23898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:19.403904 23898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:19.408674 23898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "6a09cb5f898b4f0eaef384701af8613e"
format_stamp: "Formatted at 2026-08-07 03:09:19 on dist-test-slave-tm1g"
server_key: "19ff59337dbccc859bb755f3ad88df32"
server_key_iv: "f3594bed9bd04c277a212c1bb8d9881b"
server_key_version: "encryptionkey@0"
I20260807 03:09:19.409814 23898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "6a09cb5f898b4f0eaef384701af8613e"
format_stamp: "Formatted at 2026-08-07 03:09:19 on dist-test-slave-tm1g"
server_key: "19ff59337dbccc859bb755f3ad88df32"
server_key_iv: "f3594bed9bd04c277a212c1bb8d9881b"
server_key_version: "encryptionkey@0"
I20260807 03:09:19.418490 23898 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s
I20260807 03:09:19.423918 23914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:19.426083 23898 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260807 03:09:19.426276 23898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "6a09cb5f898b4f0eaef384701af8613e"
format_stamp: "Formatted at 2026-08-07 03:09:19 on dist-test-slave-tm1g"
server_key: "19ff59337dbccc859bb755f3ad88df32"
server_key_iv: "f3594bed9bd04c277a212c1bb8d9881b"
server_key_version: "encryptionkey@0"
I20260807 03:09:19.426733 23898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:19.453823 23898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:19.460480 23898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:19.460839 23898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:19.490676 23898 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:44907
I20260807 03:09:19.490741 23967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:44907 every 8 connection(s)
I20260807 03:09:19.492784 23898 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:09:19.498730 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 23898
I20260807 03:09:19.499161 23968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:19.499179 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:09:19.500825 17532 external_mini_cluster.cc:1428] Setting key 33d573195796e6afb19d7fd987a2f518
I20260807 03:09:19.523281 23968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e: Bootstrap starting.
Aug 07 03:09:19 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM
I20260807 03:09:19.529736 23968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:19.531869 23968 log.cc:824] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:19.536973 23968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e: No bootstrap required, opened a new log
I20260807 03:09:19.542346 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:19.504402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.17.31.62:44907)
0807 03:09:19.505794 (+ 1392us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:19.505819 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:19.505896 (+ 77us) server_negotiation.cc:407] Connection header received
0807 03:09:19.509741 (+ 3845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:19.509775 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:19.510509 (+ 734us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:19.511042 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:19.514561 (+ 3519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:19.516595 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:19.519995 (+ 3400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:19.521118 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:19.521329 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:19.529518 (+ 8189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:19.529529 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:19.529553 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:19.529626 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:19.535674 (+ 6048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:19.536740 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:19.536749 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:19.536755 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:19.537018 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:19.537645 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:19.537659 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:19.537661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:19.538610 (+ 949us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:19.539243 (+ 633us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:19.540046 (+ 803us) server_negotiation.cc:299] Negotiation successful
0807 03:09:19.540708 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1092,"thread_start_us":694,"threads_started":1}
I20260807 03:09:19.551664 23968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } }
I20260807 03:09:19.552261 23968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:19.552433 23968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a09cb5f898b4f0eaef384701af8613e, State: Initialized, Role: FOLLOWER
I20260807 03:09:19.553272 23968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } }
I20260807 03:09:19.553643 23968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:09:19.553907 23968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:09:19.554207 23968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:19.558044 23968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } }
I20260807 03:09:19.558796 23968 leader_election.cc:302] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a09cb5f898b4f0eaef384701af8613e; no voters:
I20260807 03:09:19.559749 23968 leader_election.cc:288] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:09:19.559916 23973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:19.561086 23973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [term 1 LEADER]: Becoming Leader. State: Replica: 6a09cb5f898b4f0eaef384701af8613e, State: Running, Role: LEADER
I20260807 03:09:19.561949 23973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } }
I20260807 03:09:19.563329 23968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:09:19.567852 23974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a09cb5f898b4f0eaef384701af8613e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } }
I20260807 03:09:19.568490 23974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:19.568015 23975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a09cb5f898b4f0eaef384701af8613e. Latest consensus state: current_term: 1 leader_uuid: "6a09cb5f898b4f0eaef384701af8613e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } }
I20260807 03:09:19.568778 23975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:19.581768 23979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:09:19.590555 23979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:09:19.608224 23979 catalog_manager.cc:1347] Generated new cluster ID: a5db12aebde34806b4cc58fef502962e
I20260807 03:09:19.608438 23979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:09:19.638535 23979 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:09:19.641207 23979 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:09:19.651042 23979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e: Generated new TSK 0
I20260807 03:09:19.652631 23979 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.1.keytab.
Entry for principal oryx/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.1.keytab.
I20260807 03:09:19.758834 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44907
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.1.keytab
--principal=oryx/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:20.334583 23996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:20.335007 23996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:20.335153 23996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:20.345654 23996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:20.345808 23996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:20.345985 23996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:09:20.360436 23996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.1.keytab
--principal=oryx/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:20.362483 23996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:20.364603 23996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:20.383882 24005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:20.383838 24003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:20.384277 24001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:20.385507 23996 server_base.cc:1034] running on GCE node
I20260807 03:09:20.386546 23996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:20.387930 23996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:20.389248 23996 hybrid_clock.cc:648] HybridClock initialized: now 1786072160389182 us; error 73 us; skew 500 ppm
I20260807 03:09:20.389714 23996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:20 dist-test-slave-tm1g krb5kdc[23885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072160, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:20.394443 23996 init.cc:377] Logged in from keytab as oryx/127.17.31.1@KRBTEST.COM (short username oryx)
I20260807 03:09:20.396396 23996 webserver.cc:460] Webserver started at http://127.17.31.1:33865/ using document root <none> and password file <none>
I20260807 03:09:20.397384 23996 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:20.397532 23996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:20.397984 23996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:20.402369 23996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "4231fdc0f70f4ee083331c18cdeeccc0"
format_stamp: "Formatted at 2026-08-07 03:09:20 on dist-test-slave-tm1g"
server_key: "12db1f2140c1e495aaff4ecbf1db3904"
server_key_iv: "60b1a0a89150a0af899a42a65c509e40"
server_key_version: "encryptionkey@0"
I20260807 03:09:20.403616 23996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "4231fdc0f70f4ee083331c18cdeeccc0"
format_stamp: "Formatted at 2026-08-07 03:09:20 on dist-test-slave-tm1g"
server_key: "12db1f2140c1e495aaff4ecbf1db3904"
server_key_iv: "60b1a0a89150a0af899a42a65c509e40"
server_key_version: "encryptionkey@0"
I20260807 03:09:20.411810 23996 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260807 03:09:20.416814 24012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:20.418792 23996 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260807 03:09:20.419008 23996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "4231fdc0f70f4ee083331c18cdeeccc0"
format_stamp: "Formatted at 2026-08-07 03:09:20 on dist-test-slave-tm1g"
server_key: "12db1f2140c1e495aaff4ecbf1db3904"
server_key_iv: "60b1a0a89150a0af899a42a65c509e40"
server_key_version: "encryptionkey@0"
I20260807 03:09:20.419271 23996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:20.458868 23996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:20.465103 23996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:20.465574 23996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:20.467121 23996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:20.469539 23996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:20.469645 23996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:20.469769 23996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:20.469846 23996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:20.526271 23996 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:33945
I20260807 03:09:20.526398 24133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:33945 every 8 connection(s)
I20260807 03:09:20.528343 23996 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:09:20.534704 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 23996
I20260807 03:09:20.535115 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:09:20.536172 17532 external_mini_cluster.cc:1428] Setting key 38f1350b6aebcebf80d564e1dbf1132e
Aug 07 03:09:20 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072160, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM
I20260807 03:09:20.567835 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:20.532283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:59617 (local address 127.17.31.62:44907)
0807 03:09:20.533345 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:20.533352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:20.534750 (+ 1398us) server_negotiation.cc:407] Connection header received
0807 03:09:20.537995 (+ 3245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:20.538002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:20.538154 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:20.538384 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:20.542145 (+ 3761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:20.543434 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:20.547864 (+ 4430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:20.548540 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:20.548718 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:20.556105 (+ 7387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:20.556126 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:20.556135 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:20.556194 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:20.562513 (+ 6319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:20.563573 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:20.563583 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:20.563601 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:20.563869 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:20.564458 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:20.564472 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:20.564479 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:20.565178 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:20.565422 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:20.566484 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0807 03:09:20.566921 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":563,"threads_started":1}
I20260807 03:09:20.568784 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:20.533325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.1:59617)
0807 03:09:20.534521 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:20.534592 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:20.537534 (+ 2942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:20.538881 (+ 1347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:20.538899 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:20.540134 (+ 1235us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:20.541803 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:20.541865 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:20.543670 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:20.543690 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:20.547565 (+ 3875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:20.547593 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:20.548918 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:20.548929 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:20.549577 (+ 648us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:20.550975 (+ 1398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:20.551064 (+ 89us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:20.555700 (+ 4636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:20.562697 (+ 6997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:20.562704 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:20.562726 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:20.563364 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:20.564061 (+ 697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:20.564066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:20.564068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:20.564313 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:20.565362 (+ 1049us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:20.565369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:20.565880 (+ 511us) client_negotiation.cc:769] Sending connection context
0807 03:09:20.566231 (+ 351us) client_negotiation.cc:240] Negotiation successful
0807 03:09:20.566920 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":468,"threads_started":1}
I20260807 03:09:20.571319 24134 heartbeater.cc:347] Connected to a master server at 127.17.31.62:44907
I20260807 03:09:20.571767 24134 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:20.573050 24134 heartbeater.cc:511] Master 127.17.31.62:44907 requested a full tablet report, sending...
I20260807 03:09:20.576826 23931 ts_manager.cc:194] Registered new tserver with Master: 4231fdc0f70f4ee083331c18cdeeccc0 (127.17.31.1:33945)
WARNING: no policy specified for oryx/127.17.31.2@KRBTEST.COM; defaulting to no policy
I20260807 03:09:20.579546 23931 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:59617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.2.keytab.
Entry for principal oryx/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.2.keytab.
I20260807 03:09:20.618609 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44907
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.2.keytab
--principal=oryx/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:21.165562 24140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:21.166013 24140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:21.166165 24140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:21.177165 24140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:21.177315 24140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:21.177492 24140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:09:21.192541 24140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.2.keytab
--principal=oryx/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:21.194934 24140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:21.197276 24140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:21.219264 24145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:21.219651 24146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:21.219764 24148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:21.221745 24140 server_base.cc:1034] running on GCE node
I20260807 03:09:21.222812 24140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:21.224318 24140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:21.225809 24140 hybrid_clock.cc:648] HybridClock initialized: now 1786072161225715 us; error 103 us; skew 500 ppm
I20260807 03:09:21.226357 24140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:21 dist-test-slave-tm1g krb5kdc[23885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072161, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:21.236850 24140 init.cc:377] Logged in from keytab as oryx/127.17.31.2@KRBTEST.COM (short username oryx)
I20260807 03:09:21.239171 24140 webserver.cc:460] Webserver started at http://127.17.31.2:43591/ using document root <none> and password file <none>
I20260807 03:09:21.240224 24140 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:21.240322 24140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:21.240684 24140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:21.245428 24140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "87877ea3035848769a83b861efb845e7"
format_stamp: "Formatted at 2026-08-07 03:09:21 on dist-test-slave-tm1g"
server_key: "f0c1d1db58e00eab7ccc40084761fdcd"
server_key_iv: "f7192edbb6c2ad959bacb372fdb74604"
server_key_version: "encryptionkey@0"
I20260807 03:09:21.246719 24140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "87877ea3035848769a83b861efb845e7"
format_stamp: "Formatted at 2026-08-07 03:09:21 on dist-test-slave-tm1g"
server_key: "f0c1d1db58e00eab7ccc40084761fdcd"
server_key_iv: "f7192edbb6c2ad959bacb372fdb74604"
server_key_version: "encryptionkey@0"
I20260807 03:09:21.255642 24140 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260807 03:09:21.261150 24155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:21.263415 24140 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260807 03:09:21.263666 24140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "87877ea3035848769a83b861efb845e7"
format_stamp: "Formatted at 2026-08-07 03:09:21 on dist-test-slave-tm1g"
server_key: "f0c1d1db58e00eab7ccc40084761fdcd"
server_key_iv: "f7192edbb6c2ad959bacb372fdb74604"
server_key_version: "encryptionkey@0"
I20260807 03:09:21.264070 24140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:21.292703 24140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:21.299676 24140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:21.300136 24140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:21.301708 24140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:21.304133 24140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:21.304255 24140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:21.304517 24140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:21.304607 24140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:21.362700 24140 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:43243
I20260807 03:09:21.362826 24274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:43243 every 8 connection(s)
I20260807 03:09:21.364849 24140 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:09:21.366264 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 24140
I20260807 03:09:21.366714 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:09:21.367820 17532 external_mini_cluster.cc:1428] Setting key daebfbf172ca248156e66a226d4bd7e7
Aug 07 03:09:21 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072161, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM
I20260807 03:09:21.407920 24277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:21.370030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:34379 (local address 127.17.31.62:44907)
0807 03:09:21.370957 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:21.371009 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:21.373190 (+ 2181us) server_negotiation.cc:407] Connection header received
0807 03:09:21.378119 (+ 4929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:21.378140 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:21.378310 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:21.378599 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:21.382454 (+ 3855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:21.383620 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:21.387930 (+ 4310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:21.389267 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:21.389547 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:21.395035 (+ 5488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:21.395064 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:21.395083 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:21.395188 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:21.401832 (+ 6644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:21.403140 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:21.403160 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:21.403167 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:21.403464 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:21.404098 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:21.404116 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:21.404134 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:21.404864 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:21.405165 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:21.406291 (+ 1126us) server_negotiation.cc:299] Negotiation successful
0807 03:09:21.406835 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":491,"threads_started":1}
I20260807 03:09:21.408651 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:21.371451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.2:34379)
0807 03:09:21.372920 (+ 1469us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:21.373010 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:21.377600 (+ 4590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:21.379076 (+ 1476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:21.379104 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:21.380639 (+ 1535us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:21.382107 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:21.382157 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:21.383886 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:21.383910 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:21.387710 (+ 3800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:21.387748 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:21.389602 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:21.389608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:21.389934 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:21.391073 (+ 1139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:21.391117 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:21.394631 (+ 3514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:21.402113 (+ 7482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:21.402137 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:21.402150 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:21.402958 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:21.403631 (+ 673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:21.403635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:21.403638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:21.403931 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:21.405072 (+ 1141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:21.405079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:21.405630 (+ 551us) client_negotiation.cc:769] Sending connection context
0807 03:09:21.405989 (+ 359us) client_negotiation.cc:240] Negotiation successful
0807 03:09:21.406720 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"thread_start_us":645,"threads_started":1}
I20260807 03:09:21.410984 24275 heartbeater.cc:347] Connected to a master server at 127.17.31.62:44907
I20260807 03:09:21.411372 24275 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.17.31.3@KRBTEST.COM; defaulting to no policy
I20260807 03:09:21.412570 24275 heartbeater.cc:511] Master 127.17.31.62:44907 requested a full tablet report, sending...
I20260807 03:09:21.415102 23931 ts_manager.cc:194] Registered new tserver with Master: 87877ea3035848769a83b861efb845e7 (127.17.31.2:43243)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.31.3@KRBTEST.COM" created.
I20260807 03:09:21.416464 23931 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:34379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.3.keytab.
Entry for principal oryx/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.3.keytab.
I20260807 03:09:21.454402 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44907
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.3.keytab
--principal=oryx/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:09:21.584169 24134 heartbeater.cc:503] Master 127.17.31.62:44907 was elected leader, sending a full tablet report...
W20260807 03:09:21.838802 24281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:21.839054 24281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:21.839149 24281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:21.846792 24281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:21.846885 24281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:21.846992 24281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:09:21.856354 24281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.3.keytab
--principal=oryx/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:21.857476 24281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:21.858798 24281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:21.871656 24292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:21.871743 24288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:21.871743 24287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:21.872948 24281 server_base.cc:1034] running on GCE node
I20260807 03:09:21.873513 24281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:21.874439 24281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:21.875638 24281 hybrid_clock.cc:648] HybridClock initialized: now 1786072161875588 us; error 46 us; skew 500 ppm
I20260807 03:09:21.875941 24281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:21 dist-test-slave-tm1g krb5kdc[23885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072161, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:21.878960 24281 init.cc:377] Logged in from keytab as oryx/127.17.31.3@KRBTEST.COM (short username oryx)
I20260807 03:09:21.880412 24281 webserver.cc:460] Webserver started at http://127.17.31.3:33329/ using document root <none> and password file <none>
I20260807 03:09:21.881173 24281 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:21.881326 24281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:21.881628 24281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:21.884524 24281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "7710219f3ce64aa48489cf3d46c4e070"
format_stamp: "Formatted at 2026-08-07 03:09:21 on dist-test-slave-tm1g"
server_key: "0d447956329040d3b2bf3102b455d545"
server_key_iv: "fc6e937205a8af0b938c40853a01914a"
server_key_version: "encryptionkey@0"
I20260807 03:09:21.885468 24281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "7710219f3ce64aa48489cf3d46c4e070"
format_stamp: "Formatted at 2026-08-07 03:09:21 on dist-test-slave-tm1g"
server_key: "0d447956329040d3b2bf3102b455d545"
server_key_iv: "fc6e937205a8af0b938c40853a01914a"
server_key_version: "encryptionkey@0"
I20260807 03:09:21.890542 24281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 03:09:21.893855 24299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:21.895156 24281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:09:21.895315 24281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "7710219f3ce64aa48489cf3d46c4e070"
format_stamp: "Formatted at 2026-08-07 03:09:21 on dist-test-slave-tm1g"
server_key: "0d447956329040d3b2bf3102b455d545"
server_key_iv: "fc6e937205a8af0b938c40853a01914a"
server_key_version: "encryptionkey@0"
I20260807 03:09:21.895517 24281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:21.910089 24281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:21.913672 24281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:21.913954 24281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:21.914938 24281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:21.916205 24281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:21.916280 24281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:21.916445 24281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:21.916517 24281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:21.949648 24281 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:38523
I20260807 03:09:21.949695 24417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:38523 every 8 connection(s)
I20260807 03:09:21.950920 24281 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:09:21.955255 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 24281
I20260807 03:09:21.955459 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:09:21.955929 17532 external_mini_cluster.cc:1428] Setting key 276e537c18ba6af998951b289e7fff6f
Aug 07 03:09:21 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072161, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM
I20260807 03:09:21.970014 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:21.953374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:60013 (local address 127.17.31.62:44907)
0807 03:09:21.953795 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:21.953799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:21.954865 (+ 1066us) server_negotiation.cc:407] Connection header received
0807 03:09:21.956832 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:21.956838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:21.956921 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:21.957039 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:21.958503 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:21.959250 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:21.960766 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:21.961177 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:21.961308 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:21.964598 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:21.964603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:21.964606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:21.964677 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:21.967103 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:21.967783 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:21.967788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:21.967790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:21.967900 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:21.968303 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:21.968308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:21.968313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:21.968582 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:21.968743 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:21.969354 (+ 611us) server_negotiation.cc:299] Negotiation successful
0807 03:09:21.969601 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":203,"threads_started":1}
I20260807 03:09:21.971025 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:21.953909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.3:60013)
0807 03:09:21.954719 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:21.954758 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:21.956541 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:21.957263 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:21.957270 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:21.957743 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:21.958280 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:21.958295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:21.959407 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:21.959413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:21.960615 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:21.960626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:21.961392 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:21.961401 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:21.961641 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:21.962214 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:21.962235 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:21.964395 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:21.967238 (+ 2843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:21.967246 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:21.967260 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:21.967654 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:21.968003 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:21.968010 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:21.968013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:21.968188 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:21.968706 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:21.968713 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:21.969020 (+ 307us) client_negotiation.cc:769] Sending connection context
0807 03:09:21.969255 (+ 235us) client_negotiation.cc:240] Negotiation successful
0807 03:09:21.969731 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":256,"threads_started":1}
I20260807 03:09:21.972162 24418 heartbeater.cc:347] Connected to a master server at 127.17.31.62:44907
I20260807 03:09:21.972450 24418 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:21.973130 24418 heartbeater.cc:511] Master 127.17.31.62:44907 requested a full tablet report, sending...
I20260807 03:09:21.974476 23931 ts_manager.cc:194] Registered new tserver with Master: 7710219f3ce64aa48489cf3d46c4e070 (127.17.31.3:38523)
I20260807 03:09:21.975212 23931 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:60013
I20260807 03:09:21.983083 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.17.31.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.31.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.31.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.61.keytab.
Entry for principal oryx/127.17.31.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.61.keytab.
I20260807 03:09:22.031540 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.61:37419
--webserver_interface=127.17.31.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.61.keytab
--principal=oryx/127.17.31.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:44907,127.17.31.61:37419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:22.360013 24424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:22.360325 24424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:22.360489 24424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:22.367527 24424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:09:22.367653 24424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:22.367758 24424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:22.367849 24424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:09:22.367926 24424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:09:22.377324 24424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:44907,127.17.31.61:37419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.61.keytab
--principal=oryx/127.17.31.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.61:37419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.31.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
W20260807 03:09:22.377694 24424 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.17.31.62:44907,127.17.31.61:37419'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260807 03:09:22.378686 24424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:22.379902 24424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:22.391498 24433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:22.391533 24431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:22.391727 24430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:22.392647 24424 server_base.cc:1034] running on GCE node
I20260807 03:09:22.393378 24424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:22.394758 24424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:22.395992 24424 hybrid_clock.cc:648] HybridClock initialized: now 1786072162395932 us; error 53 us; skew 500 ppm
I20260807 03:09:22.396294 24424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:22 dist-test-slave-tm1g krb5kdc[23885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072162, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:22.399416 24424 init.cc:377] Logged in from keytab as oryx/127.17.31.61@KRBTEST.COM (short username oryx)
I20260807 03:09:22.400889 24424 webserver.cc:460] Webserver started at http://127.17.31.61:42731/ using document root <none> and password file <none>
I20260807 03:09:22.401516 24424 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:22.401602 24424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:22.401907 24424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:22.404881 24424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/data/instance:
uuid: "c8706df62d4147e7a32dc4b6e6d2511a"
format_stamp: "Formatted at 2026-08-07 03:09:22 on dist-test-slave-tm1g"
server_key: "d0a4ccabd8a043071608f4ba7b379082"
server_key_iv: "c6fb8fdcd637f6ce50bc31df22491bb7"
server_key_version: "encryptionkey@0"
I20260807 03:09:22.405622 24424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/wal/instance:
uuid: "c8706df62d4147e7a32dc4b6e6d2511a"
format_stamp: "Formatted at 2026-08-07 03:09:22 on dist-test-slave-tm1g"
server_key: "d0a4ccabd8a043071608f4ba7b379082"
server_key_iv: "c6fb8fdcd637f6ce50bc31df22491bb7"
server_key_version: "encryptionkey@0"
I20260807 03:09:22.410559 24424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260807 03:09:22.413669 24440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:22.414953 24424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:09:22.415086 24424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/wal
uuid: "c8706df62d4147e7a32dc4b6e6d2511a"
format_stamp: "Formatted at 2026-08-07 03:09:22 on dist-test-slave-tm1g"
server_key: "d0a4ccabd8a043071608f4ba7b379082"
server_key_iv: "c6fb8fdcd637f6ce50bc31df22491bb7"
server_key_version: "encryptionkey@0"
I20260807 03:09:22.415254 24424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:22.420408 24275 heartbeater.cc:503] Master 127.17.31.62:44907 was elected leader, sending a full tablet report...
I20260807 03:09:22.431609 24424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:22.435997 24424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:22.436275 24424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:22.453972 24424 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.61:37419
I20260807 03:09:22.453992 24493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.61:37419 every 8 connection(s)
I20260807 03:09:22.455335 24424 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/data/info.pb
I20260807 03:09:22.459053 24494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:22.461979 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 24424
I20260807 03:09:22.466471 24494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:09:22 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.31.61@KRBTEST.COM
Aug 07 03:09:22 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072162, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.61@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM
I20260807 03:09:22.485548 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:22.464180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.17.31.61:37419)
0807 03:09:22.464876 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:22.464886 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:22.464925 (+ 39us) server_negotiation.cc:407] Connection header received
0807 03:09:22.466728 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:22.466749 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:22.467137 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:22.467510 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:22.468982 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.470591 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.472170 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.473005 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.473132 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:22.478046 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:22.478055 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:22.478066 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:22.478094 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:22.481253 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:22.482057 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:22.482063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:22.482069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:22.482213 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:22.482692 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:22.482696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:22.482698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:22.483285 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:22.483633 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:22.483899 (+ 266us) server_negotiation.cc:299] Negotiation successful
0807 03:09:22.484290 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":303,"threads_started":1}
I20260807 03:09:22.485822 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:22.469276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.17.31.62:44907)
0807 03:09:22.469515 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:22.469524 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:22.471002 (+ 1478us) server_negotiation.cc:407] Connection header received
0807 03:09:22.471201 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:22.471209 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:22.471285 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:22.471405 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:22.472667 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.473598 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.475171 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.475880 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.476026 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:22.479521 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:22.479527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:22.479530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:22.479563 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:22.482930 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:22.483774 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:22.483780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:22.483784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:22.483949 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:22.484386 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:22.484391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:22.484400 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:22.484703 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:22.484820 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:22.485341 (+ 521us) server_negotiation.cc:299] Negotiation successful
0807 03:09:22.485508 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 03:09:22.485956 24498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:22.469224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.0.0.1:49488)
0807 03:09:22.470876 (+ 1652us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:22.470910 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:22.471075 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:22.471486 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:22.471493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:22.472201 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:22.472515 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:22.472528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.473749 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.473755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:22.475025 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:22.475037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.476080 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.476085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:22.476216 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:22.476803 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:22.476834 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:22.479327 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:22.483050 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:22.483058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:22.483064 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:22.483640 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:22.484081 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:22.484086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:22.484089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:22.484254 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:22.484832 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:22.484839 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:22.485019 (+ 180us) client_negotiation.cc:769] Sending connection context
0807 03:09:22.485231 (+ 212us) client_negotiation.cc:240] Negotiation successful
0807 03:09:22.485561 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1414,"thread_start_us":1275,"threads_started":1}
I20260807 03:09:22.488471 24494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:09:22 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072162, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.61@KRBTEST.COM for oryx/127.17.31.61@KRBTEST.COM
I20260807 03:09:22.501387 24498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:22.489191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:37419 (local address 127.0.0.1:47328)
0807 03:09:22.489335 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:22.489349 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:22.489434 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:22.489751 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:22.489757 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:22.490183 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:22.490378 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:22.490390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.491292 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.491297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:22.492656 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:22.492669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.493301 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.493306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:22.493483 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:22.493980 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:22.493999 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:22.496171 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:22.498953 (+ 2782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:22.498960 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:22.498963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:22.499411 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:22.499742 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:22.499748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:22.499754 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:22.499884 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:22.500425 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:22.500430 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:22.500583 (+ 153us) client_negotiation.cc:769] Sending connection context
0807 03:09:22.500716 (+ 133us) client_negotiation.cc:240] Negotiation successful
0807 03:09:22.501000 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260807 03:09:22.501571 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:22.489372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47328 (local address 127.17.31.61:37419)
0807 03:09:22.489509 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:22.489513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:22.489526 (+ 13us) server_negotiation.cc:407] Connection header received
0807 03:09:22.489568 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:22.489572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:22.489621 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:22.489693 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:22.490506 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.491160 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.492785 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.493176 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.493266 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:22.496342 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:22.496347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:22.496350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:22.496375 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:22.498828 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:22.499520 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:22.499528 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:22.499531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:22.499639 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:22.499982 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:22.499990 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:22.499993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:22.500314 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:22.500457 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:22.500797 (+ 340us) server_negotiation.cc:299] Negotiation successful
0807 03:09:22.501148 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":50}
I20260807 03:09:22.506026 24494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: Bootstrap starting.
I20260807 03:09:22.509030 24494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:22.510021 24494 log.cc:824] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:22.512503 24494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: No bootstrap required, opened a new log
I20260807 03:09:22.519385 24494 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } }
I20260807 03:09:22.519711 24494 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:22.519831 24494 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8706df62d4147e7a32dc4b6e6d2511a, State: Initialized, Role: FOLLOWER
I20260807 03:09:22.520349 24494 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } }
I20260807 03:09:22.521124 24501 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } } }
I20260807 03:09:22.521391 24501 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:09:22.522086 24494 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: configured and running, proceeding with master startup.
W20260807 03:09:22.531064 24513 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:09:22.531204 24513 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:09:22.534569 24424 master_runner.cc:428] Detected that this master c8706df62d4147e7a32dc4b6e6d2511a is joining an existing cluster
I20260807 03:09:22.534652 24424 master_runner.cc:432] Initiating AddMaster RPC to add 127.17.31.61:37419
I20260807 03:09:22.568638 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:22.538615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.17.31.61:37419)
0807 03:09:22.538819 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:22.538826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:22.551714 (+ 12888us) server_negotiation.cc:407] Connection header received
0807 03:09:22.551901 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:22.551905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:22.551974 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:22.552067 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:22.553808 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.554979 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.557411 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.558077 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.558178 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:22.561390 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:22.561395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:22.561400 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:22.561433 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:22.565414 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:22.566364 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:22.566368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:22.566371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:22.566509 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:22.566963 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:22.566968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:22.566970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:22.567321 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:22.567461 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:22.568028 (+ 567us) server_negotiation.cc:299] Negotiation successful
0807 03:09:22.568252 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 03:09:22.568627 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:22.538275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49504 (local address 127.17.31.62:44907)
0807 03:09:22.538473 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:22.538480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:22.551105 (+ 12625us) server_negotiation.cc:407] Connection header received
0807 03:09:22.552081 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:22.552086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:22.552178 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:22.552249 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:22.553825 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.554979 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.557424 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.558077 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.558181 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:22.561401 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:22.561407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:22.561411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:22.561446 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:22.565338 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:22.566377 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:22.566382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:22.566385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:22.566509 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:22.566970 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:22.566975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:22.566979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:22.567327 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:22.567445 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:22.568060 (+ 615us) server_negotiation.cc:299] Negotiation successful
0807 03:09:22.568259 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 03:09:22.569712 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:22.538941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:37419 (local address 127.0.0.1:47340)
0807 03:09:22.551619 (+ 12678us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:22.551632 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:22.551784 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:22.552233 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:22.552238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:22.553057 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:22.553655 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:22.553674 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.555227 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.555237 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:22.557266 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:22.557281 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.558542 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.558547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:22.558838 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:22.559837 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:22.559857 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:22.561194 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:22.565548 (+ 4354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:22.565554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:22.565560 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:22.566217 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:22.566647 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:22.566651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:22.566655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:22.566843 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:22.567545 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:22.567548 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:22.567752 (+ 204us) client_negotiation.cc:769] Sending connection context
0807 03:09:22.568817 (+ 1065us) client_negotiation.cc:240] Negotiation successful
0807 03:09:22.569124 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12566,"mutex_wait_us":198,"spinlock_wait_cycles":23040,"thread_start_us":12382,"threads_started":1}
I20260807 03:09:22.569792 24519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:22.538345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.0.0.1:49504)
0807 03:09:22.550954 (+ 12609us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:22.550983 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:22.551927 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:22.552449 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:22.552454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:22.553081 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:22.553683 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:22.553692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.555179 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.555186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:22.557266 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:22.557281 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.558510 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.558518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:22.558839 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:22.559775 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:22.559809 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:22.561194 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:22.565507 (+ 4313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:22.565513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:22.565520 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:22.566217 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:22.566647 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:22.566652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:22.566655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:22.566843 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:22.567539 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:22.567543 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:22.567752 (+ 209us) client_negotiation.cc:769] Sending connection context
0807 03:09:22.568883 (+ 1131us) client_negotiation.cc:240] Negotiation successful
0807 03:09:22.569303 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12399,"mutex_wait_us":495,"spinlock_wait_cycles":41216,"thread_start_us":12225,"threads_started":1}
WARNING: no policy specified for oryx/127.17.31.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.31.60@KRBTEST.COM" created.
Aug 07 03:09:22 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072159, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.62@KRBTEST.COM for oryx/127.17.31.61@KRBTEST.COM
I20260807 03:09:22.592000 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:22.577157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.17.31.61:37419)
0807 03:09:22.577430 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:22.577435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:22.577924 (+ 489us) server_negotiation.cc:407] Connection header received
0807 03:09:22.578252 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:22.578260 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:22.578403 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:22.578523 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:22.579571 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.580271 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.581657 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.582231 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.582451 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:22.585740 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:22.585746 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:22.585750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:22.585783 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:22.589189 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:22.589926 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:22.589935 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:22.589938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:22.590104 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:22.590541 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:22.590549 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:22.590554 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:22.590821 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:22.590965 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:22.591418 (+ 453us) server_negotiation.cc:299] Negotiation successful
0807 03:09:22.591634 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 03:09:22.592073 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:22.577042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:37419 (local address 127.0.0.1:47346)
0807 03:09:22.577758 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:22.577814 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:22.577997 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:22.578589 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:22.578597 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:22.579205 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:22.579413 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:22.579422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.580411 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.580417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:22.581511 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:22.581523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.582489 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.582496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:22.582686 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:22.583274 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:22.583295 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:22.585538 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:22.589327 (+ 3789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:22.589340 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:22.589348 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:22.589769 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:22.590234 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:22.590238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:22.590241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:22.590425 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:22.590960 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:22.590967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:22.591134 (+ 167us) client_negotiation.cc:769] Sending connection context
0807 03:09:22.591328 (+ 194us) client_negotiation.cc:240] Negotiation successful
0807 03:09:22.591657 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":354,"threads_started":1}
I20260807 03:09:22.594177 23931 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.17.31.61:37419: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a"
member_type: NON_VOTER
last_known_addr {
host: "127.17.31.61"
port: 37419
}
attrs {
promote: true
}
}
dest_uuid: "6a09cb5f898b4f0eaef384701af8613e"
cas_config_opid_index: -1
I20260807 03:09:22.595417 23931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: NON_VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: true } }
I20260807 03:09:22.606369 24469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:22.605110 24525 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c8706df62d4147e7a32dc4b6e6d2511a (127.17.31.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: NON_VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: true } } }
I20260807 03:09:22.609422 24525 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.17.31.61:37419
I20260807 03:09:22.610261 24469 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 FOLLOWER]: Refusing update from remote peer 6a09cb5f898b4f0eaef384701af8613e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260807 03:09:22.610437 24424 master.cc:537] Master@127.17.31.61:37419 shutting down...
I20260807 03:09:22.614957 24527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: NON_VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:22.617480 24526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6a09cb5f898b4f0eaef384701af8613e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: NON_VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: true } } }
I20260807 03:09:22.617934 24526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.31.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.60.keytab.
Entry for principal oryx/127.17.31.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.60.keytab.
I20260807 03:09:22.621889 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.60:41139
--webserver_interface=127.17.31.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.60.keytab
--principal=oryx/127.17.31.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:44907,127.17.31.61:37419,127.17.31.60:41139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:09:22.627244 24424 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:22.627720 24424 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:22.627970 24424 tablet_replica.cc:330] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: stopping tablet replica
W20260807 03:09:22.651249 23917 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260807 03:09:22.653208 23917 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e -> Peer c8706df62d4147e7a32dc4b6e6d2511a (127.17.31.61:37419): Couldn't send request to peer c8706df62d4147e7a32dc4b6e6d2511a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:09:22.657620 24424 master.cc:552] Master@127.17.31.61:37419 shutdown complete.
I20260807 03:09:22.657758 24424 master_runner.cc:305] Clearing existing system tablet
I20260807 03:09:22.658250 24424 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:22.658383 24424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:22.661422 24424 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260807 03:09:22.663105 24533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:22.663462 24424 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:22.663713 24424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260807 03:09:22.663836 24424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/wal
uuid: "c8706df62d4147e7a32dc4b6e6d2511a"
format_stamp: "Formatted at 2026-08-07 03:09:22 on dist-test-slave-tm1g"
server_key: "d0a4ccabd8a043071608f4ba7b379082"
server_key_iv: "c6fb8fdcd637f6ce50bc31df22491bb7"
server_key_version: "encryptionkey@0"
I20260807 03:09:22.668817 24424 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:22.675582 24424 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 03:09:22.675771 24424 log.cc:1196] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260807 03:09:22.676258 24424 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: Deleting consensus metadata
I20260807 03:09:22.676568 24424 master_runner.cc:315] Copying system tablet from 127.17.31.62:44907
I20260807 03:09:22.679977 24424 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: tablet copy: Beginning tablet copy session from remote peer at address 127.17.31.62:44907
I20260807 03:09:22.697785 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:22.681069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.17.31.62:44907)
0807 03:09:22.681273 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:22.681280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:22.682891 (+ 1611us) server_negotiation.cc:407] Connection header received
0807 03:09:22.683098 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:22.683104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:22.683194 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:22.683307 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:22.684372 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.685673 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.687861 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.688558 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.688691 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:22.691220 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:22.691227 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:22.691232 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:22.691275 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:22.694798 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:22.695606 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:22.695612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:22.695616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:22.695776 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:22.696210 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:22.696215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:22.696219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:22.696598 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:22.696749 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:22.697218 (+ 469us) server_negotiation.cc:299] Negotiation successful
0807 03:09:22.697451 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 03:09:22.697834 24539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:22.680986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.0.0.1:49518)
0807 03:09:22.682759 (+ 1773us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:22.682778 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:22.682942 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:22.683353 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:22.683360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:22.683897 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:22.684203 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:22.684215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.685844 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.685853 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:22.687685 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:22.687704 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:22.688741 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:22.688748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:22.689000 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:22.689807 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:22.689837 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:22.690993 (+ 1156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:22.694956 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:22.694961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:22.694966 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:22.695464 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:22.695900 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:22.695905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:22.695908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:22.696080 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:22.696729 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:22.696733 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:22.696912 (+ 179us) client_negotiation.cc:769] Sending connection context
0807 03:09:22.697073 (+ 161us) client_negotiation.cc:240] Negotiation successful
0807 03:09:22.697387 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1619,"thread_start_us":1442,"threads_started":1}
I20260807 03:09:22.699281 23953 tablet_copy_service.cc:140] P 6a09cb5f898b4f0eaef384701af8613e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c8706df62d4147e7a32dc4b6e6d2511a ({username='oryx', principal='oryx/127.17.31.61@KRBTEST.COM'} at 127.0.0.1:49518)
I20260807 03:09:22.699683 23953 tablet_copy_service.cc:161] P 6a09cb5f898b4f0eaef384701af8613e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c8706df62d4147e7a32dc4b6e6d2511a at {username='oryx', principal='oryx/127.17.31.61@KRBTEST.COM'} at 127.0.0.1:49518: session id = c8706df62d4147e7a32dc4b6e6d2511a-00000000000000000000000000000000
I20260807 03:09:22.704706 23953 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:09:22.707540 24424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 03:09:22.714773 24424 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: tablet copy: Starting download of 0 data blocks...
I20260807 03:09:22.715147 24424 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: tablet copy: Starting download of 1 WAL segments...
I20260807 03:09:22.718811 24424 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 03:09:22.723850 23953 tablet_copy_service.cc:342] P 6a09cb5f898b4f0eaef384701af8613e: Request end of tablet copy session c8706df62d4147e7a32dc4b6e6d2511a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.17.31.61@KRBTEST.COM'} at 127.0.0.1:49518
I20260807 03:09:22.724117 23953 tablet_copy_service.cc:433] P 6a09cb5f898b4f0eaef384701af8613e: ending tablet copy session c8706df62d4147e7a32dc4b6e6d2511a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c8706df62d4147e7a32dc4b6e6d2511a
I20260807 03:09:22.736099 24424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:22.738864 24540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:22.740031 24541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:22.746999 24543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:22.747435 24424 server_base.cc:1034] running on GCE node
I20260807 03:09:22.747768 24424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:22.748441 24424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:22.749749 24424 hybrid_clock.cc:648] HybridClock initialized: now 1786072162749663 us; error 41 us; skew 500 ppm
I20260807 03:09:22.750020 24424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:22 dist-test-slave-tm1g krb5kdc[23885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072162, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:22.752308 24424 init.cc:377] Logged in from keytab as oryx/127.17.31.61@KRBTEST.COM (short username oryx)
I20260807 03:09:22.757030 24424 webserver.cc:460] Webserver started at http://127.17.31.61:40297/ using document root <none> and password file <none>
I20260807 03:09:22.757426 24424 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:22.757514 24424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:22.760553 24424 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 03:09:22.762936 24550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:22.763423 24424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260807 03:09:22.763644 24424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/wal
uuid: "c8706df62d4147e7a32dc4b6e6d2511a"
format_stamp: "Formatted at 2026-08-07 03:09:22 on dist-test-slave-tm1g"
server_key: "d0a4ccabd8a043071608f4ba7b379082"
server_key_iv: "c6fb8fdcd637f6ce50bc31df22491bb7"
server_key_version: "encryptionkey@0"
I20260807 03:09:22.763810 24424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:22.790946 24424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:22.795295 24424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:22.826246 24424 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.61:37419
I20260807 03:09:22.826292 24604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.61:37419 every 8 connection(s)
I20260807 03:09:22.827451 24424 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-1/data/info.pb
I20260807 03:09:22.833178 24605 sys_catalog.cc:263] Verifying existing consensus state
I20260807 03:09:22.837695 24605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: Bootstrap starting.
I20260807 03:09:22.856506 24605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:09:22.857259 24605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: Bootstrap complete.
I20260807 03:09:22.858425 24605 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: NON_VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: true } }
I20260807 03:09:22.858794 24605 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8706df62d4147e7a32dc4b6e6d2511a, State: Initialized, Role: LEARNER
I20260807 03:09:22.859115 24605 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: NON_VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: true } }
I20260807 03:09:22.860134 24608 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: NON_VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: true } } }
I20260807 03:09:22.860404 24605 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:09:22.860553 24608 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: This master's current role is: LEARNER
I20260807 03:09:22.871863 24621 catalog_manager.cc:1259] Loaded cluster ID: a5db12aebde34806b4cc58fef502962e
I20260807 03:09:22.871989 24621 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: loading cluster ID for follower catalog manager: success
I20260807 03:09:22.875169 24621 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: acquiring CA information for follower catalog manager: success
I20260807 03:09:22.878937 24621 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 03:09:22.977662 24418 heartbeater.cc:503] Master 127.17.31.62:44907 was elected leader, sending a full tablet report...
W20260807 03:09:23.138799 24530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:23.139199 24530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:23.139359 24530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:23.148514 24530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:09:23.148675 24530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:23.148761 24530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:23.148872 24530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:09:23.148948 24530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:09:23.162015 24530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:44907,127.17.31.61:37419,127.17.31.60:41139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.60.keytab
--principal=oryx/127.17.31.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.60:41139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.31.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:23.163808 24530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:23.165540 24530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:23.180568 24626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:23.181218 24627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:23.181241 24629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:23.182631 24530 server_base.cc:1034] running on GCE node
I20260807 03:09:23.183512 24530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:23.185005 24530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:23.186276 24530 hybrid_clock.cc:648] HybridClock initialized: now 1786072163186174 us; error 45 us; skew 500 ppm
I20260807 03:09:23.186729 24530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:23 dist-test-slave-tm1g krb5kdc[23885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072163, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:23.190989 24530 init.cc:377] Logged in from keytab as oryx/127.17.31.60@KRBTEST.COM (short username oryx)
I20260807 03:09:23.193003 24530 webserver.cc:460] Webserver started at http://127.17.31.60:37005/ using document root <none> and password file <none>
I20260807 03:09:23.193881 24530 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:23.194113 24530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:23.194530 24530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:23.197950 24530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/data/instance:
uuid: "9bfc45b1240243d4b9701258adfb2723"
format_stamp: "Formatted at 2026-08-07 03:09:23 on dist-test-slave-tm1g"
server_key: "2b15e2558f7bbe7901d4fa972ebea48c"
server_key_iv: "c5de1da1db9510f670b2d7677fb0373c"
server_key_version: "encryptionkey@0"
I20260807 03:09:23.199007 24530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/wal/instance:
uuid: "9bfc45b1240243d4b9701258adfb2723"
format_stamp: "Formatted at 2026-08-07 03:09:23 on dist-test-slave-tm1g"
server_key: "2b15e2558f7bbe7901d4fa972ebea48c"
server_key_iv: "c5de1da1db9510f670b2d7677fb0373c"
server_key_version: "encryptionkey@0"
I20260807 03:09:23.205044 24530 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260807 03:09:23.209429 24637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:23.210902 24530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 03:09:23.211133 24530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/wal
uuid: "9bfc45b1240243d4b9701258adfb2723"
format_stamp: "Formatted at 2026-08-07 03:09:23 on dist-test-slave-tm1g"
server_key: "2b15e2558f7bbe7901d4fa972ebea48c"
server_key_iv: "c5de1da1db9510f670b2d7677fb0373c"
server_key_version: "encryptionkey@0"
I20260807 03:09:23.211416 24530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:23.227885 24530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:23.233034 24530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:23.233507 24530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:23.255275 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.243454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.17.31.61:37419)
0807 03:09:23.245792 (+ 2338us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:23.245798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:23.246118 (+ 320us) server_negotiation.cc:407] Connection header received
0807 03:09:23.246357 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:23.246362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:23.246461 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:23.246621 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:23.247114 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.248785 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.251623 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.253690 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.253884 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.254214 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:23.254565 (+ 351us) server_negotiation.cc:299] Negotiation successful
0807 03:09:23.254724 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2094,"thread_start_us":1866,"threads_started":1}
I20260807 03:09:23.255275 24649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.243131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:37419 (local address 127.0.0.1:47348)
0807 03:09:23.245998 (+ 2867us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:23.246015 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:23.246187 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:23.246661 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:23.246666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:23.246701 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:23.246965 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.246976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.248965 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.248977 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.251349 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.251381 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.253982 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.253990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.254381 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.254424 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 03:09:23.254536 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 03:09:23.254688 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2748,"thread_start_us":2602,"threads_started":1}
I20260807 03:09:23.256903 24579 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260807 03:09:23.257989 24608 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a09cb5f898b4f0eaef384701af8613e. Latest consensus state: current_term: 1 leader_uuid: "6a09cb5f898b4f0eaef384701af8613e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: NON_VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: true } } }
I20260807 03:09:23.258375 24608 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: This master's current role is: LEARNER
I20260807 03:09:23.274289 24530 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.60:41139
I20260807 03:09:23.274475 24694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.60:41139 every 8 connection(s)
I20260807 03:09:23.276108 24530 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/data/info.pb
I20260807 03:09:23.281078 24695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:23.281852 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 24530
I20260807 03:09:23.293394 24695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:09:23 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.31.60@KRBTEST.COM
Aug 07 03:09:23 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072163, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.60@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM
I20260807 03:09:23.316215 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.284435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52152 (local address 127.17.31.60:41139)
0807 03:09:23.285983 (+ 1548us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:23.286000 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:23.286080 (+ 80us) server_negotiation.cc:407] Connection header received
0807 03:09:23.289793 (+ 3713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:23.289821 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:23.290399 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:23.290957 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:23.292773 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.295443 (+ 2670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.297614 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.299200 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.299425 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.305344 (+ 5919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:23.305385 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:23.305396 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:23.305452 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:23.310159 (+ 4707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.311535 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.311552 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.311569 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.311843 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.312264 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.312270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.312273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.313209 (+ 936us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:23.313636 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:23.313990 (+ 354us) server_negotiation.cc:299] Negotiation successful
0807 03:09:23.314494 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1245,"thread_start_us":939,"threads_started":1}
I20260807 03:09:23.318214 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.297036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.17.31.62:44907)
0807 03:09:23.297982 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:23.297989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:23.299198 (+ 1209us) server_negotiation.cc:407] Connection header received
0807 03:09:23.299271 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:23.299274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:23.299335 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:23.299448 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:23.301274 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.302249 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.303985 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.304560 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.304659 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.309997 (+ 5338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:23.310002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:23.310005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:23.310043 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:23.314978 (+ 4935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.315816 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.315821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.315823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.316077 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.316534 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.316539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.316545 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.316804 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:23.316917 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:23.317554 (+ 637us) server_negotiation.cc:299] Negotiation successful
0807 03:09:23.317757 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":556,"threads_started":1}
I20260807 03:09:23.318280 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.297190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.0.0.1:49534)
0807 03:09:23.298828 (+ 1638us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:23.298875 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:23.299054 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:23.299550 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:23.299567 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:23.300595 (+ 1028us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:23.301081 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.301090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.302451 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.302456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.303835 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.303846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.304871 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.304881 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.305163 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.306070 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:23.306121 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:23.309773 (+ 3652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:23.315140 (+ 5367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:23.315165 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:23.315175 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:23.315669 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:23.316244 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:23.316248 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:23.316250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:23.316403 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:23.317002 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:23.317009 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:23.317220 (+ 211us) client_negotiation.cc:769] Sending connection context
0807 03:09:23.317476 (+ 256us) client_negotiation.cc:240] Negotiation successful
0807 03:09:23.317807 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1301,"thread_start_us":988,"threads_started":1}
I20260807 03:09:23.319727 24695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:09:23 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072163, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.60@KRBTEST.COM for oryx/127.17.31.61@KRBTEST.COM
I20260807 03:09:23.334483 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.320810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:37419 (local address 127.0.0.1:47360)
0807 03:09:23.320996 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:23.321038 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:23.321133 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:23.321443 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:23.321447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:23.321792 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:23.321985 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.321994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.323194 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.323197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.324386 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.324396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.325252 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.325257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.325524 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.326145 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:23.326161 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:23.328539 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:23.332118 (+ 3579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:23.332121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:23.332124 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:23.332653 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:23.333043 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:23.333046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:23.333048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:23.333200 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:23.333677 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:23.333680 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:23.333805 (+ 125us) client_negotiation.cc:769] Sending connection context
0807 03:09:23.333908 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 03:09:23.334128 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 03:09:23.334635 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.320977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47360 (local address 127.17.31.61:37419)
0807 03:09:23.321161 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:23.321170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:23.321203 (+ 33us) server_negotiation.cc:407] Connection header received
0807 03:09:23.321265 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:23.321269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:23.321326 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:23.321426 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:23.322132 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.323027 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.324542 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.325069 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.325201 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.328699 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:23.328703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:23.328706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:23.328768 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:23.332005 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.332760 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.332763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.332766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.332951 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.333288 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.333291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.333293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.333585 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:23.333692 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:23.333989 (+ 297us) server_negotiation.cc:299] Negotiation successful
0807 03:09:23.334199 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:09:23.335843 24695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:09:23 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072163, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.60@KRBTEST.COM for oryx/127.17.31.60@KRBTEST.COM
I20260807 03:09:23.350581 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.336851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:41139 (local address 127.0.0.1:52162)
0807 03:09:23.337034 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:23.337048 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:23.337171 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:23.337452 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:23.337456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:23.337982 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:23.338177 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.338185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.339340 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.339347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.340366 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.340377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.341149 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.341154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.341501 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.342089 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:23.342104 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:23.344581 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:23.348070 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:23.348076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:23.348080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:23.348409 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:23.348793 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:23.348799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:23.348803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:23.348947 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:23.349509 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:23.349521 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:23.349655 (+ 134us) client_negotiation.cc:769] Sending connection context
0807 03:09:23.349790 (+ 135us) client_negotiation.cc:240] Negotiation successful
0807 03:09:23.350047 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260807 03:09:23.350696 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.336626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.17.31.60:41139)
0807 03:09:23.336778 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:23.336785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:23.337115 (+ 330us) server_negotiation.cc:407] Connection header received
0807 03:09:23.337276 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:23.337281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:23.337343 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:23.337436 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:23.338347 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.339215 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.340498 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.340975 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.341146 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.344798 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:23.344803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:23.344806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:23.344832 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:23.347925 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.348522 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.348528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.348534 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.348685 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.349054 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.349060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.349064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.349386 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:23.349532 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:23.349880 (+ 348us) server_negotiation.cc:299] Negotiation successful
0807 03:09:23.350210 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":56}
I20260807 03:09:23.356348 24695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: Bootstrap starting.
I20260807 03:09:23.360693 24695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:23.362695 24695 log.cc:824] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:23.366588 24695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: No bootstrap required, opened a new log
I20260807 03:09:23.376217 24695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } }
I20260807 03:09:23.376611 24695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:23.376842 24695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfc45b1240243d4b9701258adfb2723, State: Initialized, Role: FOLLOWER
I20260807 03:09:23.377442 24695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } }
I20260807 03:09:23.378758 24703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } } }
I20260807 03:09:23.379267 24703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:09:23.379874 24695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 03:09:23.393916 24714 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:09:23.394101 24714 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:09:23.398407 24530 master_runner.cc:428] Detected that this master 9bfc45b1240243d4b9701258adfb2723 is joining an existing cluster
I20260807 03:09:23.398680 24530 master_runner.cc:432] Initiating AddMaster RPC to add 127.17.31.60:41139
I20260807 03:09:23.423748 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 23996
I20260807 03:09:23.440834 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.405625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49536 (local address 127.17.31.62:44907)
0807 03:09:23.405811 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:23.405817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:23.417720 (+ 11903us) server_negotiation.cc:407] Connection header received
0807 03:09:23.418088 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:23.418093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:23.418184 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:23.418258 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:23.423482 (+ 5224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.424699 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.427605 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.428388 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.428510 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.432133 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:23.432141 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:23.432147 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:23.432198 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:23.437046 (+ 4848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.438053 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.438059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.438062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.438251 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.438822 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.438828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.438831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.439198 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:23.439316 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:23.440068 (+ 752us) server_negotiation.cc:299] Negotiation successful
0807 03:09:23.440342 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 03:09:23.442216 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.405653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.17.31.60:41139)
0807 03:09:23.405997 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:23.406004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:23.409766 (+ 3762us) server_negotiation.cc:407] Connection header received
0807 03:09:23.410052 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:23.410060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:23.410148 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:23.410231 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:23.412519 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.414359 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.418237 (+ 3878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.420199 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.420356 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.428700 (+ 8344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:23.428710 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:23.428717 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:23.428770 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:23.435185 (+ 6415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.436310 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.436318 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.436324 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.436545 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.437119 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.437127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.437132 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.437716 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:23.437884 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:23.441285 (+ 3401us) server_negotiation.cc:299] Negotiation successful
0807 03:09:23.441587 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 03:09:23.442862 24722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.406909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:41139 (local address 127.0.0.1:52164)
0807 03:09:23.409585 (+ 2676us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:23.409609 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:23.409906 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:23.410501 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:23.410511 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:23.411744 (+ 1233us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:23.412335 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.412353 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.414691 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.414699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.418098 (+ 3399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.418109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.421027 (+ 2918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.421031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.421178 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.424346 (+ 3168us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:23.424366 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:23.428504 (+ 4138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:23.435368 (+ 6864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:23.435375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:23.435382 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:23.436145 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:23.436687 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:23.436694 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:23.436701 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:23.436985 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:23.437952 (+ 967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:23.437964 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:23.438214 (+ 250us) client_negotiation.cc:769] Sending connection context
0807 03:09:23.438443 (+ 229us) client_negotiation.cc:240] Negotiation successful
0807 03:09:23.438914 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2469,"thread_start_us":1945,"threads_started":1}
I20260807 03:09:23.445107 24721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.406926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.0.0.1:49536)
0807 03:09:23.417613 (+ 10687us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:23.417631 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:23.417891 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:23.418559 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:23.418564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:23.420231 (+ 1667us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:23.423297 (+ 3066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.423311 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.424985 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.424990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.427431 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.427454 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.428925 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.428932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.429189 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.430128 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:23.430150 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:23.431869 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:23.437214 (+ 5345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:23.437220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:23.437223 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:23.437861 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:23.438432 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:23.438437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:23.438440 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:23.438653 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:23.439422 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:23.439426 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:23.439680 (+ 254us) client_negotiation.cc:769] Sending connection context
0807 03:09:23.441373 (+ 1693us) client_negotiation.cc:240] Negotiation successful
0807 03:09:23.444212 (+ 2839us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10493,"mutex_wait_us":44,"spinlock_wait_cycles":83328,"thread_start_us":9312,"threads_started":1}
I20260807 03:09:23.454110 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.407063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:37419 (local address 127.0.0.1:47376)
0807 03:09:23.421890 (+ 14827us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:23.421909 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:23.422075 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:23.422494 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:23.422497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:23.424172 (+ 1675us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:23.425637 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.425651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.427714 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.427722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.440399 (+ 12677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.440422 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.442193 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.442211 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.442487 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.443415 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:23.443441 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:23.445144 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:23.450504 (+ 5360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:23.450508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:23.450511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:23.451302 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:23.451791 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:23.451797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:23.451800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:23.452059 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:23.452850 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:23.452855 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:23.453185 (+ 330us) client_negotiation.cc:769] Sending connection context
0807 03:09:23.453330 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 03:09:23.453629 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14671,"spinlock_wait_cycles":95104,"thread_start_us":13641,"threads_started":1}
I20260807 03:09:23.455253 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.405653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47376 (local address 127.17.31.61:37419)
0807 03:09:23.405821 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:23.405827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:23.422007 (+ 16180us) server_negotiation.cc:407] Connection header received
0807 03:09:23.422208 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:23.422215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:23.422349 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:23.422463 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:23.425828 (+ 3365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.427493 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.440679 (+ 13186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.441541 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.441676 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.445429 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:23.445437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:23.445443 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:23.445522 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:23.450291 (+ 4769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.451451 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.451458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.451461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.451657 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.452192 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.452197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.452199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.452641 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:23.452764 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:23.454594 (+ 1830us) server_negotiation.cc:299] Negotiation successful
0807 03:09:23.454852 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Aug 07 03:09:23 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.31.1@KRBTEST.COM
Aug 07 03:09:23 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072159, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.62@KRBTEST.COM for oryx/127.17.31.60@KRBTEST.COM
I20260807 03:09:23.469173 24723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.425773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57606 (local address 127.17.31.1:33945)
0807 03:09:23.441123 (+ 15350us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:23.441134 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:23.441174 (+ 40us) server_negotiation.cc:407] Connection header received
0807 03:09:23.441261 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:23.441270 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:23.441608 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:23.441835 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:23.445963 (+ 4128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.448311 (+ 2348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.451440 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.452499 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.458805 (+ 6306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.460584 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:23.460596 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:23.460609 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:23.460642 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:23.465646 (+ 5004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.466428 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.466437 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.466441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.466622 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.467145 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.467150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.467154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.467503 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:23.467905 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:23.468290 (+ 385us) server_negotiation.cc:299] Negotiation successful
0807 03:09:23.468644 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15100,"thread_start_us":16349,"threads_started":1}
I20260807 03:09:23.471071 24128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:23.476882 24649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.452728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:41139 (local address 127.0.0.1:52180)
0807 03:09:23.453204 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:23.453223 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:23.453324 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:23.454281 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:23.454285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:23.454918 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:23.455212 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.455227 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.457174 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.457179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.458413 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.458430 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.459540 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.459547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.459808 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.460487 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:23.460509 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:23.463750 (+ 3241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:23.469533 (+ 5783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:23.469539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:23.469543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:23.470180 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:23.474789 (+ 4609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:23.474792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:23.474795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:23.475049 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:23.475640 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:23.475645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:23.475907 (+ 262us) client_negotiation.cc:769] Sending connection context
0807 03:09:23.476048 (+ 141us) client_negotiation.cc:240] Negotiation successful
0807 03:09:23.476363 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278}
I20260807 03:09:23.476927 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.453508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.17.31.60:41139)
0807 03:09:23.453770 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:23.453777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:23.453793 (+ 16us) server_negotiation.cc:407] Connection header received
0807 03:09:23.453877 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:23.453882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:23.453954 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:23.454043 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:23.455443 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.456979 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.458567 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.459362 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.459503 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.464047 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:23.464055 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:23.464059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:23.464097 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:23.469330 (+ 5233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.470360 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.470365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.470369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.474583 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.475178 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.475183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.475186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.475514 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:23.475675 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:23.476148 (+ 473us) server_negotiation.cc:299] Negotiation successful
0807 03:09:23.476437 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260807 03:09:23.478762 23931 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.17.31.60:41139: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9bfc45b1240243d4b9701258adfb2723"
member_type: NON_VOTER
last_known_addr {
host: "127.17.31.60"
port: 41139
}
attrs {
promote: true
}
}
dest_uuid: "6a09cb5f898b4f0eaef384701af8613e"
cas_config_opid_index: 5
I20260807 03:09:23.479797 23931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: NON_VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: true } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true } }
I20260807 03:09:23.482074 24645 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9bfc45b1240243d4b9701258adfb2723 (127.17.31.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: NON_VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: true } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true } } }
W20260807 03:09:23.486161 24644 tablet.cc:2369] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:09:23.488185 24579 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 LEARNER]: Refusing update from remote peer 6a09cb5f898b4f0eaef384701af8613e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 03:09:23.488929 24670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:23.493078 24645 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.17.31.60:41139
I20260807 03:09:23.502274 24726 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: NON_VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260807 03:09:23.507018 24579 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9bfc45b1240243d4b9701258adfb2723 (127.17.31.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: NON_VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: true } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true } } }
I20260807 03:09:23.511355 24608 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a09cb5f898b4f0eaef384701af8613e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: NON_VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: true } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true } } }
I20260807 03:09:23.511755 24608 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: This master's current role is: LEARNER
I20260807 03:09:23.512276 24725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6a09cb5f898b4f0eaef384701af8613e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: NON_VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: true } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true } } }
I20260807 03:09:23.512679 24725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:23.519074 24725 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e: attempting to promote NON_VOTER c8706df62d4147e7a32dc4b6e6d2511a to VOTER
I20260807 03:09:23.520171 24725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true } }
I20260807 03:09:23.523904 24579 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 LEARNER]: Refusing update from remote peer 6a09cb5f898b4f0eaef384701af8613e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260807 03:09:23.524820 24728 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 03:09:23.526790 24670 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 1 FOLLOWER]: Refusing update from remote peer 6a09cb5f898b4f0eaef384701af8613e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 03:09:23.527338 24530 master.cc:537] Master@127.17.31.60:41139 shutting down...
I20260807 03:09:23.528234 24728 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8706df62d4147e7a32dc4b6e6d2511a (127.17.31.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true } } }
I20260807 03:09:23.529099 24669 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 1 FOLLOWER]: Refusing update from remote peer 6a09cb5f898b4f0eaef384701af8613e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260807 03:09:23.528806 24579 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c8706df62d4147e7a32dc4b6e6d2511a (127.17.31.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true } } }
I20260807 03:09:23.531985 24725 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:23.533442 24645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6a09cb5f898b4f0eaef384701af8613e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true } } }
I20260807 03:09:23.533736 24645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:23.534055 24645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a09cb5f898b4f0eaef384701af8613e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true } } }
I20260807 03:09:23.534359 24645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:23.535378 24608 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a09cb5f898b4f0eaef384701af8613e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true } } }
I20260807 03:09:23.535750 24608 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:09:23.542212 24530 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:23.542786 24530 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:23.543084 24530 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: stopping tablet replica
W20260807 03:09:23.548416 23916 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e -> Peer 9bfc45b1240243d4b9701258adfb2723 (127.17.31.60:41139): Couldn't send request to peer 9bfc45b1240243d4b9701258adfb2723. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 03:09:23.550724 24530 master.cc:552] Master@127.17.31.60:41139 shutdown complete.
I20260807 03:09:23.550822 24530 master_runner.cc:305] Clearing existing system tablet
I20260807 03:09:23.551168 24530 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:23.551295 24530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:23.553444 24530 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260807 03:09:23.554883 24735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:23.555186 24530 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:23.555467 24530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260807 03:09:23.555577 24530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/wal
uuid: "9bfc45b1240243d4b9701258adfb2723"
format_stamp: "Formatted at 2026-08-07 03:09:23 on dist-test-slave-tm1g"
server_key: "2b15e2558f7bbe7901d4fa972ebea48c"
server_key_iv: "c5de1da1db9510f670b2d7677fb0373c"
server_key_version: "encryptionkey@0"
I20260807 03:09:23.558979 24530 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:23.564082 24530 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 03:09:23.564260 24530 log.cc:1196] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260807 03:09:23.564766 24530 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: Deleting consensus metadata
I20260807 03:09:23.565028 24530 master_runner.cc:315] Copying system tablet from 127.17.31.62:44907
I20260807 03:09:23.574918 24530 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: tablet copy: Beginning tablet copy session from remote peer at address 127.17.31.62:44907
I20260807 03:09:23.591475 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.575778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49546 (local address 127.17.31.62:44907)
0807 03:09:23.576154 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:23.576164 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:23.576952 (+ 788us) server_negotiation.cc:407] Connection header received
0807 03:09:23.577183 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:23.577188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:23.577297 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:23.577414 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:23.578567 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.579333 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.581337 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.582058 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.582197 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.584794 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:23.584805 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:23.584809 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:23.584841 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:23.588603 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.589292 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.589299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.589302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.589431 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:23.589901 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:23.589908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:23.589911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:23.590188 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:23.590390 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:23.590769 (+ 379us) server_negotiation.cc:299] Negotiation successful
0807 03:09:23.591021 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260807 03:09:23.591651 24742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.575735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.0.0.1:49546)
0807 03:09:23.576835 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:23.576852 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:23.577045 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:23.577438 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:23.577442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:23.578039 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:23.578417 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.578428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.579509 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.579515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.581195 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.581207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.582241 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.582247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:23.582583 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:23.583379 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:23.583397 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:23.584533 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:23.588732 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:23.588739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:23.588743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:23.589174 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:23.589545 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:23.589550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:23.589553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:23.589778 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:23.590343 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:23.590348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:23.590534 (+ 186us) client_negotiation.cc:769] Sending connection context
0807 03:09:23.590678 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 03:09:23.591022 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"thread_start_us":803,"threads_started":1}
I20260807 03:09:23.592579 23953 tablet_copy_service.cc:140] P 6a09cb5f898b4f0eaef384701af8613e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9bfc45b1240243d4b9701258adfb2723 ({username='oryx', principal='oryx/127.17.31.60@KRBTEST.COM'} at 127.0.0.1:49546)
I20260807 03:09:23.592852 23953 tablet_copy_service.cc:161] P 6a09cb5f898b4f0eaef384701af8613e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9bfc45b1240243d4b9701258adfb2723 at {username='oryx', principal='oryx/127.17.31.60@KRBTEST.COM'} at 127.0.0.1:49546: session id = 9bfc45b1240243d4b9701258adfb2723-00000000000000000000000000000000
I20260807 03:09:23.595925 23953 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 03:09:23.597611 24530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 03:09:23.603981 24530 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: tablet copy: Starting download of 0 data blocks...
I20260807 03:09:23.604266 24530 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: tablet copy: Starting download of 1 WAL segments...
I20260807 03:09:23.606814 24530 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 03:09:23.610730 23953 tablet_copy_service.cc:342] P 6a09cb5f898b4f0eaef384701af8613e: Request end of tablet copy session 9bfc45b1240243d4b9701258adfb2723-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.17.31.60@KRBTEST.COM'} at 127.0.0.1:49546
I20260807 03:09:23.610950 23953 tablet_copy_service.cc:433] P 6a09cb5f898b4f0eaef384701af8613e: ending tablet copy session 9bfc45b1240243d4b9701258adfb2723-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9bfc45b1240243d4b9701258adfb2723
I20260807 03:09:23.621624 24530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:23.624529 24743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:23.630692 24744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:23.635480 24530 server_base.cc:1034] running on GCE node
W20260807 03:09:23.635700 24746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:23.636260 24530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:23.637116 24530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:23.638473 24530 hybrid_clock.cc:648] HybridClock initialized: now 1786072163638386 us; error 43 us; skew 500 ppm
I20260807 03:09:23.638840 24530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:23 dist-test-slave-tm1g krb5kdc[23885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072163, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:23.641359 24530 init.cc:377] Logged in from keytab as oryx/127.17.31.60@KRBTEST.COM (short username oryx)
I20260807 03:09:23.649292 24530 webserver.cc:460] Webserver started at http://127.17.31.60:32869/ using document root <none> and password file <none>
I20260807 03:09:23.649765 24530 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:23.649962 24530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:23.653985 24530 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 03:09:23.658504 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 23996
I20260807 03:09:23.664290 24753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:23.664956 24530 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.007s sys 0.003s
I20260807 03:09:23.665125 24530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/wal
uuid: "9bfc45b1240243d4b9701258adfb2723"
format_stamp: "Formatted at 2026-08-07 03:09:23 on dist-test-slave-tm1g"
server_key: "2b15e2558f7bbe7901d4fa972ebea48c"
server_key_iv: "c5de1da1db9510f670b2d7677fb0373c"
server_key_version: "encryptionkey@0"
I20260807 03:09:23.665431 24530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:23.678925 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:33945
--local_ip_for_outbound_sockets=127.17.31.1
--tserver_master_addrs=127.17.31.62:44907
--webserver_port=33865
--webserver_interface=127.17.31.1
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.1.keytab
--principal=oryx/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.31.62:44907,127.17.31.61:37419
--tserver_master_addrs=127.17.31.62:44907,127.17.31.61:37419,127.17.31.60:41139 with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:09:23.704762 24530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:23.708936 24530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:23.730873 24530 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.60:41139
I20260807 03:09:23.730942 24807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.60:41139 every 8 connection(s)
I20260807 03:09:23.732111 24530 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/master-2/data/info.pb
I20260807 03:09:23.738390 24809 sys_catalog.cc:263] Verifying existing consensus state
I20260807 03:09:23.742208 24809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: Bootstrap starting.
I20260807 03:09:23.761129 24809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:09:23.761760 24809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: Bootstrap complete.
I20260807 03:09:23.762869 24809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true } }
I20260807 03:09:23.763177 24809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9bfc45b1240243d4b9701258adfb2723, State: Initialized, Role: LEARNER
I20260807 03:09:23.763485 24809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true } }
I20260807 03:09:23.764561 24809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:09:23.764384 24812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true } } }
I20260807 03:09:23.764917 24812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [sys.catalog]: This master's current role is: LEARNER
I20260807 03:09:23.777808 24823 catalog_manager.cc:1259] Loaded cluster ID: a5db12aebde34806b4cc58fef502962e
I20260807 03:09:23.777911 24823 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: loading cluster ID for follower catalog manager: success
I20260807 03:09:23.781368 24823 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: acquiring CA information for follower catalog manager: success
I20260807 03:09:23.783669 24823 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 03:09:24.003562 24824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.994373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:41139 (local address 127.0.0.1:52192)
0807 03:09:23.994982 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:23.995008 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:23.995157 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:23.995813 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:23.995816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:23.995852 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:23.996034 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:23.996041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.998351 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:23.998356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:24.000350 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:24.000363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.002808 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.002813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:24.002983 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:24.003008 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 03:09:24.003073 (+ 65us) client_negotiation.cc:240] Negotiation successful
0807 03:09:24.003162 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":276,"threads_started":1}
I20260807 03:09:24.003723 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:23.994508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.17.31.60:41139)
0807 03:09:23.995356 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:23.995362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:23.995385 (+ 23us) server_negotiation.cc:407] Connection header received
0807 03:09:23.995491 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:23.995505 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:23.995636 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:23.995816 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:23.996195 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:23.998137 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.000541 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.002557 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.002741 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:24.003066 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:24.003132 (+ 66us) server_negotiation.cc:299] Negotiation successful
0807 03:09:24.003308 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":485,"threads_started":1}
I20260807 03:09:24.005494 24782 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260807 03:09:24.006454 24812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a09cb5f898b4f0eaef384701af8613e. Latest consensus state: current_term: 1 leader_uuid: "6a09cb5f898b4f0eaef384701af8613e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true } } }
I20260807 03:09:24.006801 24812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [sys.catalog]: This master's current role is: LEARNER
W20260807 03:09:24.089174 24758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:24.089542 24758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:24.089715 24758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:24.099047 24758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:24.099166 24758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:24.099288 24758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:09:24.111092 24758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.1.keytab
--principal=oryx/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:33945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=33865
--tserver_master_addrs=127.17.31.62:44907,127.17.31.61:37419,127.17.31.60:41139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:24.112540 24758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:24.114010 24758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:24.129577 24833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:24.129599 24836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:24.129839 24830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:24.131865 24758 server_base.cc:1034] running on GCE node
I20260807 03:09:24.132884 24758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:24.134106 24758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:24.135387 24758 hybrid_clock.cc:648] HybridClock initialized: now 1786072164135300 us; error 60 us; skew 500 ppm
I20260807 03:09:24.135793 24758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:24 dist-test-slave-tm1g krb5kdc[23885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072164, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:24.140065 24758 init.cc:377] Logged in from keytab as oryx/127.17.31.1@KRBTEST.COM (short username oryx)
I20260807 03:09:24.141937 24758 webserver.cc:460] Webserver started at http://127.17.31.1:33865/ using document root <none> and password file <none>
I20260807 03:09:24.142863 24758 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:24.143107 24758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:24.149765 24758 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260807 03:09:24.153282 24843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:24.154817 24758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 03:09:24.155046 24758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "4231fdc0f70f4ee083331c18cdeeccc0"
format_stamp: "Formatted at 2026-08-07 03:09:20 on dist-test-slave-tm1g"
server_key: "12db1f2140c1e495aaff4ecbf1db3904"
server_key_iv: "60b1a0a89150a0af899a42a65c509e40"
server_key_version: "encryptionkey@0"
I20260807 03:09:24.156358 24758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:24.180430 24758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:24.185528 24758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:24.186005 24758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:24.187487 24758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:24.189527 24758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:24.189742 24758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:24.189960 24758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:24.190100 24758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:24.247053 24758 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:33945
I20260807 03:09:24.247141 24959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:33945 every 8 connection(s)
I20260807 03:09:24.248347 24758 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:09:24.255930 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 24758
I20260807 03:09:24.256315 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 24140
Aug 07 03:09:24 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072164, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM
Aug 07 03:09:24 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072164, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.61@KRBTEST.COM
Aug 07 03:09:24 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072164, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.60@KRBTEST.COM
Aug 07 03:09:24 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.31.2@KRBTEST.COM
I20260807 03:09:24.288216 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:24.251444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:42001 (local address 127.17.31.60:41139)
0807 03:09:24.251672 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:24.251681 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:24.259465 (+ 7784us) server_negotiation.cc:407] Connection header received
0807 03:09:24.266793 (+ 7328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:24.266798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:24.266882 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:24.266959 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:24.270381 (+ 3422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.271439 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.273745 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.274501 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.274601 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:24.280485 (+ 5884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:24.280495 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:24.280501 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:24.280543 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:24.284164 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:24.285158 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:24.285166 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:24.285171 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:24.285322 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:24.285859 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:24.285866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:24.285873 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:24.286297 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:24.286453 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:24.287556 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0807 03:09:24.287803 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 03:09:24.288693 24966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:24.251476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:56903 (local address 127.17.31.62:44907)
0807 03:09:24.264791 (+ 13315us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:24.264798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:24.266367 (+ 1569us) server_negotiation.cc:407] Connection header received
0807 03:09:24.266750 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:24.266755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:24.266832 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:24.266924 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:24.269665 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.270792 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.273595 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.274191 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.274281 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:24.280964 (+ 6683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:24.280973 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:24.280981 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:24.281041 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:24.284765 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:24.285659 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:24.285665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:24.285668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:24.285825 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:24.286433 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:24.286438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:24.286440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:24.286865 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:24.287025 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:24.287639 (+ 614us) server_negotiation.cc:299] Negotiation successful
0807 03:09:24.287839 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13195,"thread_start_us":13353,"threads_started":1}
I20260807 03:09:24.289503 24967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:24.252428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:41139 (local address 127.17.31.1:42001)
0807 03:09:24.259262 (+ 6834us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:24.259324 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:24.266438 (+ 7114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:24.267443 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:24.267461 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:24.268476 (+ 1015us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:24.270166 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:24.270180 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.272033 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.272039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:24.273614 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:24.273626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.274874 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.274880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:24.275139 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:24.276281 (+ 1142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:24.276302 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:24.280227 (+ 3925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:24.284350 (+ 4123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:24.284359 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:24.284375 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:24.285016 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:24.285481 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:24.285485 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:24.285488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:24.285731 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:24.286596 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:24.286605 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:24.287082 (+ 477us) client_negotiation.cc:769] Sending connection context
0807 03:09:24.288436 (+ 1354us) client_negotiation.cc:240] Negotiation successful
0807 03:09:24.288734 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":211,"spinlock_wait_cycles":10752,"thread_start_us":2478,"threads_started":1}
I20260807 03:09:24.289934 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:24.257845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41082 (local address 127.17.31.2:43243)
0807 03:09:24.262401 (+ 4556us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:24.262411 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:24.265665 (+ 3254us) server_negotiation.cc:407] Connection header received
0807 03:09:24.265888 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:24.265900 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:24.266180 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:24.266535 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:24.269372 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.270974 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.272971 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.273799 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.273896 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:24.282043 (+ 8147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:24.282057 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:24.282071 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:24.282120 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:24.286096 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:24.286959 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:24.286968 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:24.286971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:24.287126 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:24.287638 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:24.287643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:24.287647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:24.288036 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:24.288376 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:24.289045 (+ 669us) server_negotiation.cc:299] Negotiation successful
0807 03:09:24.289319 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4295,"thread_start_us":3954,"threads_started":1}
I20260807 03:09:24.290182 24965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:24.251190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:53281 (local address 127.17.31.61:37419)
0807 03:09:24.269482 (+ 18292us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:24.269488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:24.269534 (+ 46us) server_negotiation.cc:407] Connection header received
0807 03:09:24.269594 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:24.269601 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:24.269693 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:24.269804 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:24.271236 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.272414 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.274771 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.275468 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.275592 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:24.279907 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:24.279912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:24.279915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:24.279946 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:24.284339 (+ 4393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:24.285358 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:24.285363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:24.285367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:24.285507 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:24.285969 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:24.285975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:24.285982 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:24.286379 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:24.286533 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:24.289486 (+ 2953us) server_negotiation.cc:299] Negotiation successful
0807 03:09:24.289726 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18161,"thread_start_us":18019,"threads_started":1}
I20260807 03:09:24.290884 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:24.253889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:37419 (local address 127.17.31.1:53281)
0807 03:09:24.268591 (+ 14702us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:24.268608 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:24.268710 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:24.269855 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:24.269860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:24.270626 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:24.271006 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:24.271067 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.272598 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.272603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:24.274586 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:24.274607 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.275665 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.275673 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:24.275897 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:24.276578 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:24.276604 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:24.279678 (+ 3074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:24.284503 (+ 4825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:24.284508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:24.284513 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:24.285183 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:24.285648 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:24.285653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:24.285656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:24.285832 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:24.286625 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:24.286629 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:24.287078 (+ 449us) client_negotiation.cc:769] Sending connection context
0807 03:09:24.287424 (+ 346us) client_negotiation.cc:240] Negotiation successful
0807 03:09:24.287915 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14581,"mutex_wait_us":4,"spinlock_wait_cycles":12672,"thread_start_us":12087,"threads_started":1}
I20260807 03:09:24.291903 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:24.253105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.1:56903)
0807 03:09:24.266247 (+ 13142us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:24.266265 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:24.266464 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:24.267443 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:24.267454 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:24.268930 (+ 1476us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:24.269429 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:24.269462 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.271042 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.271048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:24.273423 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:24.273436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.274645 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.274654 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:24.275066 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:24.276073 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:24.276108 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:24.280755 (+ 4647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:24.284924 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:24.284929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:24.284932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:24.285527 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:24.285966 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:24.285970 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:24.285973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:24.286276 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:24.287121 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:24.287124 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:24.287304 (+ 180us) client_negotiation.cc:769] Sending connection context
0807 03:09:24.289152 (+ 1848us) client_negotiation.cc:240] Negotiation successful
0807 03:09:24.289524 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13000,"mutex_wait_us":65,"spinlock_wait_cycles":30848,"thread_start_us":15187,"threads_started":1}
I20260807 03:09:24.292181 24270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:24.292055 24961 heartbeater.cc:347] Connected to a master server at 127.17.31.60:41139
I20260807 03:09:24.292658 24961 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:24.292837 24963 heartbeater.cc:347] Connected to a master server at 127.17.31.61:37419
I20260807 03:09:24.292989 24963 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:24.293844 24963 heartbeater.cc:511] Master 127.17.31.61:37419 requested a full tablet report, sending...
I20260807 03:09:24.294641 24962 heartbeater.cc:347] Connected to a master server at 127.17.31.62:44907
I20260807 03:09:24.294807 24962 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:24.295660 24961 heartbeater.cc:511] Master 127.17.31.60:41139 requested a full tablet report, sending...
I20260807 03:09:24.295753 24962 heartbeater.cc:511] Master 127.17.31.62:44907 requested a full tablet report, sending...
I20260807 03:09:24.297034 23931 ts_manager.cc:194] Re-registered known tserver with Master: 4231fdc0f70f4ee083331c18cdeeccc0 (127.17.31.1:33945)
I20260807 03:09:24.297181 24568 ts_manager.cc:194] Registered new tserver with Master: 4231fdc0f70f4ee083331c18cdeeccc0 (127.17.31.1:33945)
I20260807 03:09:24.297832 24772 ts_manager.cc:194] Registered new tserver with Master: 4231fdc0f70f4ee083331c18cdeeccc0 (127.17.31.1:33945)
I20260807 03:09:24.298460 23931 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:56903
I20260807 03:09:24.381593 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 24140
I20260807 03:09:24.400296 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:43243
--local_ip_for_outbound_sockets=127.17.31.2
--tserver_master_addrs=127.17.31.62:44907
--webserver_port=43591
--webserver_interface=127.17.31.2
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.2.keytab
--principal=oryx/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.31.62:44907,127.17.31.61:37419
--tserver_master_addrs=127.17.31.62:44907,127.17.31.61:37419,127.17.31.60:41139 with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:09:24.472640 24731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a09cb5f898b4f0eaef384701af8613e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: NON_VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: true } } }
I20260807 03:09:24.473248 24731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:24.474628 24731 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e: attempting to promote NON_VOTER 9bfc45b1240243d4b9701258adfb2723 to VOTER
I20260807 03:09:24.475227 24731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } }
I20260807 03:09:24.477072 24579 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 FOLLOWER]: Refusing update from remote peer 6a09cb5f898b4f0eaef384701af8613e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260807 03:09:24.477221 24782 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 1 LEARNER]: Refusing update from remote peer 6a09cb5f898b4f0eaef384701af8613e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260807 03:09:24.477696 24974 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 03:09:24.478094 24975 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260807 03:09:24.481338 24579 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9bfc45b1240243d4b9701258adfb2723 (127.17.31.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } } }
I20260807 03:09:24.480651 24974 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9bfc45b1240243d4b9701258adfb2723 (127.17.31.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } } }
I20260807 03:09:24.490406 24979 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a09cb5f898b4f0eaef384701af8613e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } } }
I20260807 03:09:24.490749 24979 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:09:24.493852 24731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6a09cb5f898b4f0eaef384701af8613e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } } }
I20260807 03:09:24.494170 24731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:24.489920 24975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a09cb5f898b4f0eaef384701af8613e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } } }
I20260807 03:09:24.494900 24975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:24.495903 24731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a09cb5f898b4f0eaef384701af8613e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } } }
I20260807 03:09:24.496302 24731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:24.496146 24782 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9bfc45b1240243d4b9701258adfb2723 (127.17.31.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } } }
I20260807 03:09:24.500465 24812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a09cb5f898b4f0eaef384701af8613e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } } }
I20260807 03:09:24.500707 24812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 03:09:24.733057 24973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:24.733340 24973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:24.733510 24973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:24.741247 24973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:24.741387 24973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:24.741571 24973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:09:24.751214 24973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.2.keytab
--principal=oryx/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:43243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=43591
--tserver_master_addrs=127.17.31.62:44907,127.17.31.61:37419,127.17.31.60:41139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:24.752439 24973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:24.753693 24973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:24.767239 24986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:24.767272 24987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:24.767339 24990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:24.768373 24973 server_base.cc:1034] running on GCE node
I20260807 03:09:24.768975 24973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:24.769853 24973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:24.771085 24973 hybrid_clock.cc:648] HybridClock initialized: now 1786072164771045 us; error 53 us; skew 500 ppm
I20260807 03:09:24.771440 24973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:24 dist-test-slave-tm1g krb5kdc[23885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072164, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:24.774603 24973 init.cc:377] Logged in from keytab as oryx/127.17.31.2@KRBTEST.COM (short username oryx)
I20260807 03:09:24.776069 24973 webserver.cc:460] Webserver started at http://127.17.31.2:43591/ using document root <none> and password file <none>
I20260807 03:09:24.776748 24973 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:24.776916 24973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:24.782398 24973 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:09:24.785050 24997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:24.786401 24973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:09:24.786568 24973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "87877ea3035848769a83b861efb845e7"
format_stamp: "Formatted at 2026-08-07 03:09:21 on dist-test-slave-tm1g"
server_key: "f0c1d1db58e00eab7ccc40084761fdcd"
server_key_iv: "f7192edbb6c2ad959bacb372fdb74604"
server_key_version: "encryptionkey@0"
I20260807 03:09:24.787518 24973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:24.799876 24973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:24.803886 24973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:24.804170 24973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:24.805081 24973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:24.806447 24973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:24.806563 24973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:24.806691 24973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:24.806777 24973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:24.841219 24973 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:43243
I20260807 03:09:24.841254 25120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:43243 every 8 connection(s)
I20260807 03:09:24.842782 24973 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:09:24.852093 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 24973
I20260807 03:09:24.852501 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 24281
Aug 07 03:09:24 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072164, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.61@KRBTEST.COM
I20260807 03:09:24.876011 25127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:24.846435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:38265 (local address 127.17.31.61:37419)
0807 03:09:24.848432 (+ 1997us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:24.848442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:24.854354 (+ 5912us) server_negotiation.cc:407] Connection header received
0807 03:09:24.856799 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:24.856805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:24.856919 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:24.857171 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:24.859011 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.859945 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.862380 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.863007 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.863147 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:24.868564 (+ 5417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:24.868569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:24.868572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:24.868640 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:24.872277 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:24.873174 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:24.873179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:24.873182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:24.873359 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:24.873914 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:24.873919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:24.873923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:24.874257 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:24.874504 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:24.875232 (+ 728us) server_negotiation.cc:299] Negotiation successful
0807 03:09:24.875523 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1838,"thread_start_us":1720,"threads_started":1}
I20260807 03:09:24.877226 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:24.847061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:37419 (local address 127.17.31.2:38265)
0807 03:09:24.850001 (+ 2940us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:24.850045 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:24.856672 (+ 6627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:24.857396 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:24.857406 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:24.858030 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:24.858838 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:24.858856 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.860109 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.860117 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:24.862216 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:24.862228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.863300 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.863310 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:24.863783 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:24.864713 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:24.864745 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:24.868285 (+ 3540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:24.872422 (+ 4137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:24.872431 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:24.872445 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:24.872975 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:24.873508 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:24.873511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:24.873513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:24.873787 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:24.874436 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:24.874444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:24.874826 (+ 382us) client_negotiation.cc:769] Sending connection context
0807 03:09:24.875098 (+ 272us) client_negotiation.cc:240] Negotiation successful
0807 03:09:24.875710 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2643,"thread_start_us":2368,"threads_started":1}
I20260807 03:09:24.879279 25123 heartbeater.cc:347] Connected to a master server at 127.17.31.61:37419
I20260807 03:09:24.879709 25123 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:24.881726 25123 heartbeater.cc:511] Master 127.17.31.61:37419 requested a full tablet report, sending...
I20260807 03:09:24.883692 24568 ts_manager.cc:194] Registered new tserver with Master: 87877ea3035848769a83b861efb845e7 (127.17.31.2:43243)
Aug 07 03:09:24 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072164, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.60@KRBTEST.COM
Aug 07 03:09:24 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.31.3@KRBTEST.COM
Aug 07 03:09:24 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072164, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM
I20260807 03:09:24.903434 25131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:24.847985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:41139 (local address 127.17.31.2:39537)
0807 03:09:24.873765 (+ 25780us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:24.873787 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:24.874081 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:24.877270 (+ 3189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:24.877275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:24.879553 (+ 2278us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:24.880398 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:24.880421 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.885591 (+ 5170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.885598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:24.887447 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:24.887459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.888358 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.888364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:24.888777 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:24.889653 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:24.889689 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:24.895066 (+ 5377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:24.899933 (+ 4867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:24.899938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:24.899942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:24.900812 (+ 870us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:24.901342 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:24.901348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:24.901352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:24.901561 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:24.902391 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:24.902399 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:24.902578 (+ 179us) client_negotiation.cc:769] Sending connection context
0807 03:09:24.902690 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 03:09:24.902908 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25588,"mutex_wait_us":68,"thread_start_us":26175,"threads_started":1}
I20260807 03:09:24.903808 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:24.846263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:39537 (local address 127.17.31.60:41139)
0807 03:09:24.876751 (+ 30488us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:24.876760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:24.876783 (+ 23us) server_negotiation.cc:407] Connection header received
0807 03:09:24.876851 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:24.876859 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:24.876950 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:24.877194 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:24.882710 (+ 5516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.885225 (+ 2515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.887590 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.888082 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.888219 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:24.895264 (+ 7045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:24.895268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:24.895272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:24.895302 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:24.899743 (+ 4441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:24.900984 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:24.900998 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:24.901002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:24.901202 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:24.901713 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:24.901719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:24.901723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:24.902218 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:24.902415 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:24.902925 (+ 510us) server_negotiation.cc:299] Negotiation successful
0807 03:09:24.903188 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30336,"thread_start_us":31075,"threads_started":1}
I20260807 03:09:24.905045 25121 heartbeater.cc:347] Connected to a master server at 127.17.31.60:41139
I20260807 03:09:24.905251 25121 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:24.906136 25129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:24.847691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.2:43555)
0807 03:09:24.855623 (+ 7932us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:24.855653 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:24.856327 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:24.884801 (+ 28474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:24.884806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:24.886091 (+ 1285us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:24.886608 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:24.886621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.888362 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.888366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:24.890870 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:24.890896 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.892417 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.892428 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:24.892680 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:24.893462 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:24.893491 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:24.897142 (+ 3651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:24.902473 (+ 5331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:24.902476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:24.902479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:24.903212 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:24.903835 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:24.903840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:24.903844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:24.904067 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:24.904729 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:24.904735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:24.904977 (+ 242us) client_negotiation.cc:769] Sending connection context
0807 03:09:24.905160 (+ 183us) client_negotiation.cc:240] Negotiation successful
0807 03:09:24.905533 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7761,"mutex_wait_us":3,"thread_start_us":7543,"threads_started":1}
I20260807 03:09:24.906402 25121 heartbeater.cc:511] Master 127.17.31.60:41139 requested a full tablet report, sending...
I20260807 03:09:24.906500 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:24.846009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:43555 (local address 127.17.31.62:44907)
0807 03:09:24.884210 (+ 38201us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:24.884216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:24.884235 (+ 19us) server_negotiation.cc:407] Connection header received
0807 03:09:24.884367 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:24.884372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:24.884479 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:24.884781 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:24.886780 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.888184 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.891070 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.892112 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.892275 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:24.897521 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:24.897536 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:24.897541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:24.897578 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:24.902355 (+ 4777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:24.903415 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:24.903421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:24.903432 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:24.903591 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:24.904205 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:24.904222 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:24.904226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:24.904591 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:24.904776 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:24.905384 (+ 608us) server_negotiation.cc:299] Negotiation successful
0807 03:09:24.905713 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38063,"thread_start_us":37946,"threads_started":1}
I20260807 03:09:24.907732 25122 heartbeater.cc:347] Connected to a master server at 127.17.31.62:44907
I20260807 03:09:24.907908 25122 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:24.908283 24772 ts_manager.cc:194] Registered new tserver with Master: 87877ea3035848769a83b861efb845e7 (127.17.31.2:43243)
I20260807 03:09:24.908311 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:24.854409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.17.31.3:38523)
0807 03:09:24.878326 (+ 23917us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:24.878338 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:24.879346 (+ 1008us) server_negotiation.cc:407] Connection header received
0807 03:09:24.879554 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:24.879563 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:24.879870 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:24.880354 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:24.884206 (+ 3852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.886132 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.888386 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:24.889098 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:24.889241 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:24.897753 (+ 8512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:24.897764 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:24.897776 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:24.897807 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:24.903384 (+ 5577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:24.904975 (+ 1591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:24.904984 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:24.904987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:24.905219 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:24.905742 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:24.905750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:24.905756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:24.906583 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:24.906906 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:24.907364 (+ 458us) server_negotiation.cc:299] Negotiation successful
0807 03:09:24.907637 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23661,"thread_start_us":23432,"threads_started":1}
I20260807 03:09:24.908730 25122 heartbeater.cc:511] Master 127.17.31.62:44907 requested a full tablet report, sending...
I20260807 03:09:24.910032 23931 ts_manager.cc:194] Re-registered known tserver with Master: 87877ea3035848769a83b861efb845e7 (127.17.31.2:43243)
I20260807 03:09:24.910247 24413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:24.911516 23931 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:43555
I20260807 03:09:24.997298 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 24281
I20260807 03:09:25.015239 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:38523
--local_ip_for_outbound_sockets=127.17.31.3
--tserver_master_addrs=127.17.31.62:44907
--webserver_port=33329
--webserver_interface=127.17.31.3
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.3.keytab
--principal=oryx/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.31.62:44907,127.17.31.61:37419
--tserver_master_addrs=127.17.31.62:44907,127.17.31.61:37419,127.17.31.60:41139 with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:09:25.301280 24962 heartbeater.cc:503] Master 127.17.31.62:44907 was elected leader, sending a full tablet report...
W20260807 03:09:25.348951 25136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:25.349191 25136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:25.349282 25136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:25.356302 25136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:25.356398 25136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:25.356511 25136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:09:25.366977 25136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.3.keytab
--principal=oryx/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:38523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=33329
--tserver_master_addrs=127.17.31.62:44907,127.17.31.61:37419,127.17.31.60:41139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:25.368234 25136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:25.369557 25136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:25.383353 25141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:25.383496 25142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:25.383630 25144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:25.384855 25136 server_base.cc:1034] running on GCE node
I20260807 03:09:25.385492 25136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:25.386473 25136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:25.387694 25136 hybrid_clock.cc:648] HybridClock initialized: now 1786072165387657 us; error 49 us; skew 500 ppm
I20260807 03:09:25.388007 25136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:25 dist-test-slave-tm1g krb5kdc[23885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072165, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:25.391153 25136 init.cc:377] Logged in from keytab as oryx/127.17.31.3@KRBTEST.COM (short username oryx)
I20260807 03:09:25.392560 25136 webserver.cc:460] Webserver started at http://127.17.31.3:33329/ using document root <none> and password file <none>
I20260807 03:09:25.393168 25136 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:25.393282 25136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:25.398283 25136 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:09:25.400885 25151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:25.402472 25136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 03:09:25.402640 25136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "7710219f3ce64aa48489cf3d46c4e070"
format_stamp: "Formatted at 2026-08-07 03:09:21 on dist-test-slave-tm1g"
server_key: "0d447956329040d3b2bf3102b455d545"
server_key_iv: "fc6e937205a8af0b938c40853a01914a"
server_key_version: "encryptionkey@0"
I20260807 03:09:25.403570 25136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:25.414855 25136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:25.418906 25136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:25.419211 25136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:25.420101 25136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:25.421375 25136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:25.421485 25136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:25.421613 25136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:25.421712 25136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:25.454234 25136 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:38523
I20260807 03:09:25.454368 25268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:38523 every 8 connection(s)
I20260807 03:09:25.455690 25136 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:09:25.457526 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 25136
Aug 07 03:09:25 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072165, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.60@KRBTEST.COM
Aug 07 03:09:25 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072165, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM
Aug 07 03:09:25 dist-test-slave-tm1g krb5kdc[23885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072165, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.61@KRBTEST.COM
I20260807 03:09:25.504446 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.458707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:32917 (local address 127.17.31.60:41139)
0807 03:09:25.477267 (+ 18560us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:25.477275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:25.477298 (+ 23us) server_negotiation.cc:407] Connection header received
0807 03:09:25.477358 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:25.477363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:25.477446 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:25.477551 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:25.479631 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.480832 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.483898 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.484533 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.484632 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.489759 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:25.489772 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:25.489779 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:25.489822 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:25.493531 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:25.494631 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:25.494636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:25.494639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:25.494899 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:25.495624 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:25.495628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:25.495631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:25.496127 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:25.496302 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:25.503677 (+ 7375us) server_negotiation.cc:299] Negotiation successful
0807 03:09:25.503980 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18430,"thread_start_us":18334,"threads_started":1}
I20260807 03:09:25.505482 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.459425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.3:42051)
0807 03:09:25.469848 (+ 10423us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:25.469917 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:25.474478 (+ 4561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:25.477039 (+ 2561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:25.477054 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:25.477925 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:25.479408 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.479434 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.481048 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.481054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.483729 (+ 2675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.483743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.484768 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.484777 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.485229 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.486361 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:25.486409 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:25.489796 (+ 3387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:25.494607 (+ 4811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:25.494615 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:25.494621 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:25.495396 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:25.499109 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:25.499117 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:25.499123 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:25.499382 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:25.501821 (+ 2439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:25.501829 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:25.502088 (+ 259us) client_negotiation.cc:769] Sending connection context
0807 03:09:25.502496 (+ 408us) client_negotiation.cc:240] Negotiation successful
0807 03:09:25.503392 (+ 896us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10007,"mutex_wait_us":366,"thread_start_us":11285,"threads_started":1}
I20260807 03:09:25.506799 25279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.459708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:41139 (local address 127.17.31.3:32917)
0807 03:09:25.476100 (+ 16392us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:25.476115 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:25.476204 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:25.477733 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:25.477742 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:25.478766 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:25.479407 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.479434 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.481048 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.481054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.483728 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.483746 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.484925 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.484933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.485222 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.486182 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:25.486216 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:25.489469 (+ 3253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:25.493700 (+ 4231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:25.493709 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:25.493725 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:25.494477 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:25.495094 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:25.495099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:25.495104 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:25.495448 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:25.496457 (+ 1009us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:25.496470 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:25.503097 (+ 6627us) client_negotiation.cc:769] Sending connection context
0807 03:09:25.505025 (+ 1928us) client_negotiation.cc:240] Negotiation successful
0807 03:09:25.505334 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16282,"mutex_wait_us":359,"thread_start_us":16132,"threads_started":1}
I20260807 03:09:25.508447 25276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.459050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:42051 (local address 127.17.31.62:44907)
0807 03:09:25.476516 (+ 17466us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:25.476523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:25.476539 (+ 16us) server_negotiation.cc:407] Connection header received
0807 03:09:25.476589 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:25.476592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:25.476651 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:25.476778 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:25.479632 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.480832 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.483874 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.484523 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.484650 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.489997 (+ 5347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:25.490005 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:25.490025 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:25.490066 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:25.494483 (+ 4417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:25.498639 (+ 4156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:25.498649 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:25.498655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:25.498963 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:25.501199 (+ 2236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:25.501208 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:25.501213 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:25.501668 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:25.501859 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:25.506967 (+ 5108us) server_negotiation.cc:299] Negotiation successful
0807 03:09:25.507727 (+ 760us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17372,"thread_start_us":17290,"threads_started":1}
I20260807 03:09:25.509341 25269 heartbeater.cc:347] Connected to a master server at 127.17.31.60:41139
I20260807 03:09:25.509740 25269 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:25.511220 25269 heartbeater.cc:511] Master 127.17.31.60:41139 requested a full tablet report, sending...
I20260807 03:09:25.513747 24772 ts_manager.cc:194] Registered new tserver with Master: 7710219f3ce64aa48489cf3d46c4e070 (127.17.31.3:38523)
I20260807 03:09:25.514719 25270 heartbeater.cc:347] Connected to a master server at 127.17.31.62:44907
I20260807 03:09:25.514927 25270 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:25.515919 25270 heartbeater.cc:511] Master 127.17.31.62:44907 requested a full tablet report, sending...
I20260807 03:09:25.517400 23931 ts_manager.cc:194] Re-registered known tserver with Master: 7710219f3ce64aa48489cf3d46c4e070 (127.17.31.3:38523)
I20260807 03:09:25.519160 23931 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:42051
I20260807 03:09:25.525960 25280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.463639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:37419 (local address 127.17.31.3:53419)
0807 03:09:25.478020 (+ 14381us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:25.478043 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:25.478449 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:25.479392 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:25.479400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:25.480133 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:25.480519 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.480534 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.482609 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.482617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.487001 (+ 4384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.487031 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.489180 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.489187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.489396 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.494806 (+ 5410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:25.494829 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:25.499995 (+ 5166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:25.521777 (+ 21782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:25.521783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:25.521786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:25.522584 (+ 798us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:25.523239 (+ 655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:25.523245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:25.523250 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:25.523529 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:25.524393 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:25.524399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:25.524650 (+ 251us) client_negotiation.cc:769] Sending connection context
0807 03:09:25.524823 (+ 173us) client_negotiation.cc:240] Negotiation successful
0807 03:09:25.525238 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14218,"mutex_wait_us":1,"thread_start_us":14060,"threads_started":1}
I20260807 03:09:25.527851 25281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.464114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:53419 (local address 127.17.31.61:37419)
0807 03:09:25.477519 (+ 13405us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:25.477528 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:25.478381 (+ 853us) server_negotiation.cc:407] Connection header received
0807 03:09:25.478717 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:25.478726 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:25.478846 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:25.479166 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:25.480834 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.482385 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.487201 (+ 4816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.488708 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.488859 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.500732 (+ 11873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:25.500742 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:25.500749 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:25.500862 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:25.521655 (+ 20793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:25.522787 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:25.522796 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:25.522803 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:25.523078 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:25.523712 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:25.523719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:25.523723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:25.524167 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:25.526709 (+ 2542us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:25.526892 (+ 183us) server_negotiation.cc:299] Negotiation successful
0807 03:09:25.527169 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13257,"thread_start_us":13169,"threads_started":1}
I20260807 03:09:25.528986 25271 heartbeater.cc:347] Connected to a master server at 127.17.31.61:37419
I20260807 03:09:25.529194 25271 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:25.530004 25271 heartbeater.cc:511] Master 127.17.31.61:37419 requested a full tablet report, sending...
I20260807 03:09:25.531597 24568 ts_manager.cc:194] Registered new tserver with Master: 7710219f3ce64aa48489cf3d46c4e070 (127.17.31.3:38523)
I20260807 03:09:25.538355 25286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.495605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49562 (local address 127.17.31.62:44907)
0807 03:09:25.510143 (+ 14538us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:25.510151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:25.510174 (+ 23us) server_negotiation.cc:407] Connection header received
0807 03:09:25.510240 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:25.510248 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:25.510368 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:25.514133 (+ 3765us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:25.514459 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.516094 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.521820 (+ 5726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.522757 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.522901 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.528325 (+ 5424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:25.528335 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:25.528339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:25.528422 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:25.533978 (+ 5556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:25.535469 (+ 1491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:25.535477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:25.535488 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:25.535690 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:25.536121 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:25.536126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:25.536130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:25.536445 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:25.536788 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:25.537679 (+ 891us) server_negotiation.cc:299] Negotiation successful
0807 03:09:25.537901 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14362,"thread_start_us":22014,"threads_started":1}
I20260807 03:09:25.551932 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.519857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52200 (local address 127.17.31.60:41139)
0807 03:09:25.520051 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:25.520056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:25.528717 (+ 8661us) server_negotiation.cc:407] Connection header received
0807 03:09:25.528933 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:25.528938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:25.529153 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:25.529248 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:25.532788 (+ 3540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.533996 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.536553 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.537512 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.537642 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.542922 (+ 5280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:25.542927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:25.542930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:25.542980 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:25.547911 (+ 4931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:25.549098 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:25.549104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:25.549108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:25.549285 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:25.549825 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:25.549831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:25.549835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:25.550341 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:25.550722 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:25.551109 (+ 387us) server_negotiation.cc:299] Negotiation successful
0807 03:09:25.551390 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 03:09:25.552209 25281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.497240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47386 (local address 127.17.31.61:37419)
0807 03:09:25.532466 (+ 35226us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:25.532471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:25.532488 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:09:25.532542 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:25.532547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:25.532646 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:25.533534 (+ 888us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:25.535726 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.536884 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.539168 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.539943 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.540081 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.544680 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:25.544685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:25.544689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:25.544717 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:25.548363 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:25.549246 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:25.549252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:25.549255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:25.549447 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:25.549888 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:25.549893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:25.549896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:25.550244 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:25.550671 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:25.551438 (+ 767us) server_negotiation.cc:299] Negotiation successful
0807 03:09:25.551757 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35093,"thread_start_us":35367,"threads_started":1}
I20260807 03:09:25.544584 23931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:09:25.557821 23931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:09:25.608676 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.584121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42556 (local address 127.17.31.3:38523)
0807 03:09:25.596295 (+ 12174us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:25.596306 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:25.596363 (+ 57us) server_negotiation.cc:407] Connection header received
0807 03:09:25.596501 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:25.596525 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:25.596996 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:25.597491 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:25.600151 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.602067 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.605433 (+ 3366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.607400 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.607540 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.607981 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:25.608099 (+ 118us) server_negotiation.cc:299] Negotiation successful
0807 03:09:25.608280 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11929,"thread_start_us":13799,"threads_started":1}
I20260807 03:09:25.609230 25301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.582869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:43243 (local address 127.0.0.1:41084)
0807 03:09:25.590464 (+ 7595us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:25.590491 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:25.591536 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:25.599735 (+ 8199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:25.599739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:25.599772 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:25.600600 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.600609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.603076 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.603081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.606092 (+ 3011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.606114 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.608251 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.608261 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.608456 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.608494 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 03:09:25.608568 (+ 74us) client_negotiation.cc:240] Negotiation successful
0807 03:09:25.608703 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7298,"thread_start_us":1026,"threads_started":1}
I20260807 03:09:25.609637 25302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.583223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.17.31.2:43243)
0807 03:09:25.598739 (+ 15516us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:25.598747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:25.598772 (+ 25us) server_negotiation.cc:407] Connection header received
0807 03:09:25.598857 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:25.598864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:25.599170 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:25.599745 (+ 575us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:25.600867 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.602900 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.606279 (+ 3379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.607996 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.608884 (+ 888us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.609088 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:25.609156 (+ 68us) server_negotiation.cc:299] Negotiation successful
0807 03:09:25.609332 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15329,"thread_start_us":15181,"threads_started":1}
I20260807 03:09:25.609817 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.583928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:38523 (local address 127.0.0.1:42556)
0807 03:09:25.596008 (+ 12080us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:25.596028 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:25.596267 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:25.598810 (+ 2543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:25.598814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:25.598856 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:25.599372 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.599387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.602440 (+ 3053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.602447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.605277 (+ 2830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.605303 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.607603 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.607610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.607778 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.607812 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 03:09:25.607886 (+ 74us) client_negotiation.cc:240] Negotiation successful
0807 03:09:25.608078 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11854,"mutex_wait_us":31,"thread_start_us":14427,"threads_started":1}
I20260807 03:09:25.610193 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.584320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:33945 (local address 127.0.0.1:57618)
0807 03:09:25.597252 (+ 12932us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:25.597277 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:25.597578 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:25.599620 (+ 2042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:25.599627 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:25.599659 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:25.600979 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.600993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.603312 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.603319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.606225 (+ 2906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.606238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.608665 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.608675 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.609178 (+ 503us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.609216 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 03:09:25.609346 (+ 130us) client_negotiation.cc:240] Negotiation successful
0807 03:09:25.609469 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12740,"mutex_wait_us":2,"thread_start_us":12450,"threads_started":1}
I20260807 03:09:25.610847 25307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.585000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57618 (local address 127.17.31.1:33945)
0807 03:09:25.598479 (+ 13479us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:25.598490 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:25.598523 (+ 33us) server_negotiation.cc:407] Connection header received
0807 03:09:25.598662 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:25.598670 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:25.598901 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:25.599298 (+ 397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:25.601217 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.603116 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.606374 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.607809 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.607935 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.608134 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:25.609666 (+ 1532us) server_negotiation.cc:299] Negotiation successful
0807 03:09:25.610477 (+ 811us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13166,"thread_start_us":13026,"threads_started":1}
I20260807 03:09:25.613355 25201 tablet_service.cc:1467] Processing CreateTablet for tablet 5e795ba096914c3897867a4160bd2349 (DEFAULT_TABLE table=test-table [id=d5ecc0ef96ee4de5b9746487595e76f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:25.613354 25048 tablet_service.cc:1467] Processing CreateTablet for tablet 5e795ba096914c3897867a4160bd2349 (DEFAULT_TABLE table=test-table [id=d5ecc0ef96ee4de5b9746487595e76f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:25.614801 25048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e795ba096914c3897867a4160bd2349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:25.614835 25201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e795ba096914c3897867a4160bd2349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:25.624933 24893 tablet_service.cc:1467] Processing CreateTablet for tablet 5e795ba096914c3897867a4160bd2349 (DEFAULT_TABLE table=test-table [id=d5ecc0ef96ee4de5b9746487595e76f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:25.626708 24893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e795ba096914c3897867a4160bd2349. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:25.629871 25308 tablet_bootstrap.cc:492] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7: Bootstrap starting.
I20260807 03:09:25.633534 25309 tablet_bootstrap.cc:492] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070: Bootstrap starting.
I20260807 03:09:25.635294 25308 tablet_bootstrap.cc:654] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:25.638172 25308 log.cc:824] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:25.639299 25309 tablet_bootstrap.cc:654] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:25.641003 25309 log.cc:824] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:25.642262 25308 tablet_bootstrap.cc:492] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7: No bootstrap required, opened a new log
I20260807 03:09:25.642613 25308 ts_tablet_manager.cc:1397] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260807 03:09:25.644686 25309 tablet_bootstrap.cc:492] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070: No bootstrap required, opened a new log
I20260807 03:09:25.645134 25309 ts_tablet_manager.cc:1397] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260807 03:09:25.645360 25313 tablet_bootstrap.cc:492] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0: Bootstrap starting.
I20260807 03:09:25.650532 25313 tablet_bootstrap.cc:654] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:25.652737 25313 log.cc:824] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:25.653386 25308 raft_consensus.cc:348] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } } peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } }
I20260807 03:09:25.654125 25308 raft_consensus.cc:374] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:25.654321 25308 raft_consensus.cc:729] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87877ea3035848769a83b861efb845e7, State: Initialized, Role: FOLLOWER
I20260807 03:09:25.655164 25308 consensus_queue.cc:260] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } } peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } }
I20260807 03:09:25.656880 25313 tablet_bootstrap.cc:492] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0: No bootstrap required, opened a new log
I20260807 03:09:25.657241 25313 ts_tablet_manager.cc:1397] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260807 03:09:25.657557 25308 ts_tablet_manager.cc:1428] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260807 03:09:25.657999 25122 heartbeater.cc:503] Master 127.17.31.62:44907 was elected leader, sending a full tablet report...
I20260807 03:09:25.661921 25309 raft_consensus.cc:348] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } } peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } }
I20260807 03:09:25.662716 25309 raft_consensus.cc:374] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:25.662978 25309 raft_consensus.cc:729] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7710219f3ce64aa48489cf3d46c4e070, State: Initialized, Role: FOLLOWER
I20260807 03:09:25.663739 25309 consensus_queue.cc:260] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } } peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } }
I20260807 03:09:25.666170 25270 heartbeater.cc:503] Master 127.17.31.62:44907 was elected leader, sending a full tablet report...
I20260807 03:09:25.666460 25309 ts_tablet_manager.cc:1428] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070: Time spent starting tablet: real 0.021s user 0.009s sys 0.010s
I20260807 03:09:25.674688 25313 raft_consensus.cc:348] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } } peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } }
I20260807 03:09:25.675163 25313 raft_consensus.cc:374] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:25.675326 25313 raft_consensus.cc:729] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4231fdc0f70f4ee083331c18cdeeccc0, State: Initialized, Role: FOLLOWER
I20260807 03:09:25.675872 25313 consensus_queue.cc:260] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } } peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } }
I20260807 03:09:25.677503 25313 ts_tablet_manager.cc:1428] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0: Time spent starting tablet: real 0.020s user 0.012s sys 0.003s
W20260807 03:09:25.726193 25273 tablet.cc:2369] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:09:25.757682 24964 tablet.cc:2369] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:09:25.792829 25315 raft_consensus.cc:482] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:09:25.793327 25315 raft_consensus.cc:504] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } } peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } }
I20260807 03:09:25.795503 25315 leader_election.cc:288] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4231fdc0f70f4ee083331c18cdeeccc0 (127.17.31.1:33945), 7710219f3ce64aa48489cf3d46c4e070 (127.17.31.3:38523)
I20260807 03:09:25.796450 25316 raft_consensus.cc:482] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:09:25.796866 25316 raft_consensus.cc:504] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } } peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } }
I20260807 03:09:25.806890 25316 leader_election.cc:288] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87877ea3035848769a83b861efb845e7 (127.17.31.2:43243), 4231fdc0f70f4ee083331c18cdeeccc0 (127.17.31.1:33945)
I20260807 03:09:25.813586 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.795787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:33945 (local address 127.17.31.2:43415)
0807 03:09:25.797754 (+ 1967us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:25.797777 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:25.798102 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:25.798844 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:25.798849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:25.798881 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:25.800555 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.800586 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.802479 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.802484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.807109 (+ 4625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.807123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.812340 (+ 5217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.812347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.812585 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.812619 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 03:09:25.812740 (+ 121us) client_negotiation.cc:240] Negotiation successful
0807 03:09:25.812874 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1780,"mutex_wait_us":1,"thread_start_us":1364,"threads_started":1}
I20260807 03:09:25.813812 25307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.796232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:43415 (local address 127.17.31.1:33945)
0807 03:09:25.796495 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:25.796502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:25.797895 (+ 1393us) server_negotiation.cc:407] Connection header received
0807 03:09:25.798298 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:25.798329 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:25.798480 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:25.798563 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:25.800805 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.802225 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.807361 (+ 5136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.809050 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.809192 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.809432 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:25.813214 (+ 3782us) server_negotiation.cc:299] Negotiation successful
0807 03:09:25.813465 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":74}
I20260807 03:09:25.815217 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.796073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:50671 (local address 127.17.31.3:38523)
0807 03:09:25.796380 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:25.796387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:25.798334 (+ 1947us) server_negotiation.cc:407] Connection header received
0807 03:09:25.798668 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:25.798673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:25.798747 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:25.798810 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:25.801426 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.804155 (+ 2729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.808777 (+ 4622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.810122 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.810195 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.810275 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:25.814737 (+ 4462us) server_negotiation.cc:299] Negotiation successful
0807 03:09:25.814914 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 03:09:25.816413 25319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.796005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:38523 (local address 127.17.31.2:50671)
0807 03:09:25.798204 (+ 2199us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:25.798223 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:25.798483 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:25.799376 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:25.799380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:25.799402 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:25.801060 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.801113 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.804387 (+ 3274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.804394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.808559 (+ 4165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.808574 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.814197 (+ 5623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.814204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.814511 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.814550 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 03:09:25.815637 (+ 1087us) client_negotiation.cc:240] Negotiation successful
0807 03:09:25.815791 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2016,"thread_start_us":2619,"threads_started":1}
I20260807 03:09:25.818540 25302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.804768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:46281 (local address 127.17.31.2:43243)
0807 03:09:25.805206 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:25.805214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:25.805242 (+ 28us) server_negotiation.cc:407] Connection header received
0807 03:09:25.805338 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:25.805351 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:25.805438 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:25.805612 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:25.806617 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.812041 (+ 5424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.814675 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.816689 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.816835 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.816951 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:25.818028 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0807 03:09:25.818165 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260807 03:09:25.818653 25280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.801584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:43243 (local address 127.17.31.3:46281)
0807 03:09:25.801957 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:25.802015 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:25.802192 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:25.805855 (+ 3663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:25.805868 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:25.805904 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:25.806276 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.806293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.812237 (+ 5944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.812245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.814463 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.814488 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.817108 (+ 2620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.817116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.817486 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.817534 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 03:09:25.817662 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 03:09:25.817826 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":36}
I20260807 03:09:25.819031 25222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e795ba096914c3897867a4160bd2349" candidate_uuid: "87877ea3035848769a83b861efb845e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7710219f3ce64aa48489cf3d46c4e070" is_pre_election: true
I20260807 03:09:25.819980 25320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.804051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:54697 (local address 127.17.31.1:33945)
0807 03:09:25.806990 (+ 2939us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:25.806996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:25.807042 (+ 46us) server_negotiation.cc:407] Connection header received
0807 03:09:25.807156 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:25.807163 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:25.807251 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:25.808094 (+ 843us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:25.808728 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.810574 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.814616 (+ 4042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.816540 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.817754 (+ 1214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.817921 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:25.819274 (+ 1353us) server_negotiation.cc:299] Negotiation successful
0807 03:09:25.819405 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":2759,"thread_start_us":2559,"threads_started":1}
I20260807 03:09:25.819938 25222 raft_consensus.cc:2454] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87877ea3035848769a83b861efb845e7 in term 0.
I20260807 03:09:25.819911 25069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e795ba096914c3897867a4160bd2349" candidate_uuid: "7710219f3ce64aa48489cf3d46c4e070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87877ea3035848769a83b861efb845e7" is_pre_election: true
I20260807 03:09:25.820724 25279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.802966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:33945 (local address 127.17.31.3:54697)
0807 03:09:25.803610 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:25.803629 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:25.803809 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:25.807490 (+ 3681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:25.807496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:25.807533 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:25.807882 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.807894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.810814 (+ 2920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.810820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.814384 (+ 3564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:25.814399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.817280 (+ 2881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.817290 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:25.818023 (+ 733us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.818056 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 03:09:25.818141 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 03:09:25.818890 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":905}
I20260807 03:09:25.820523 25069 raft_consensus.cc:2454] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7710219f3ce64aa48489cf3d46c4e070 in term 0.
I20260807 03:09:25.820773 24914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e795ba096914c3897867a4160bd2349" candidate_uuid: "87877ea3035848769a83b861efb845e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" is_pre_election: true
I20260807 03:09:25.821377 25001 leader_election.cc:302] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7710219f3ce64aa48489cf3d46c4e070, 87877ea3035848769a83b861efb845e7; no voters:
I20260807 03:09:25.821851 25152 leader_election.cc:302] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7710219f3ce64aa48489cf3d46c4e070, 87877ea3035848769a83b861efb845e7; no voters:
I20260807 03:09:25.822014 25315 raft_consensus.cc:2781] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:09:25.821795 24914 raft_consensus.cc:2454] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87877ea3035848769a83b861efb845e7 in term 0.
I20260807 03:09:25.822229 25315 raft_consensus.cc:482] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:09:25.821693 24913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e795ba096914c3897867a4160bd2349" candidate_uuid: "7710219f3ce64aa48489cf3d46c4e070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" is_pre_election: true
I20260807 03:09:25.822371 25315 raft_consensus.cc:3037] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:25.822482 25316 raft_consensus.cc:2781] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:09:25.822721 25316 raft_consensus.cc:482] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:09:25.822492 24913 raft_consensus.cc:2454] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7710219f3ce64aa48489cf3d46c4e070 in term 0.
I20260807 03:09:25.822899 25316 raft_consensus.cc:3037] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:25.826975 25315 raft_consensus.cc:504] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } } peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } }
I20260807 03:09:25.827394 25316 raft_consensus.cc:504] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } } peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } }
I20260807 03:09:25.828034 25315 leader_election.cc:288] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [CANDIDATE]: Term 1 election: Requested vote from peers 4231fdc0f70f4ee083331c18cdeeccc0 (127.17.31.1:33945), 7710219f3ce64aa48489cf3d46c4e070 (127.17.31.3:38523)
I20260807 03:09:25.828786 25069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e795ba096914c3897867a4160bd2349" candidate_uuid: "7710219f3ce64aa48489cf3d46c4e070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87877ea3035848769a83b861efb845e7"
I20260807 03:09:25.828822 24914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e795ba096914c3897867a4160bd2349" candidate_uuid: "7710219f3ce64aa48489cf3d46c4e070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4231fdc0f70f4ee083331c18cdeeccc0"
I20260807 03:09:25.828967 24913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e795ba096914c3897867a4160bd2349" candidate_uuid: "87877ea3035848769a83b861efb845e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4231fdc0f70f4ee083331c18cdeeccc0"
I20260807 03:09:25.829069 24914 raft_consensus.cc:3037] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:25.829253 25069 raft_consensus.cc:2379] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7710219f3ce64aa48489cf3d46c4e070 in current term 1: Already voted for candidate 87877ea3035848769a83b861efb845e7 in this term.
I20260807 03:09:25.830031 25222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e795ba096914c3897867a4160bd2349" candidate_uuid: "87877ea3035848769a83b861efb845e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7710219f3ce64aa48489cf3d46c4e070"
I20260807 03:09:25.830369 25222 raft_consensus.cc:2379] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 87877ea3035848769a83b861efb845e7 in current term 1: Already voted for candidate 7710219f3ce64aa48489cf3d46c4e070 in this term.
I20260807 03:09:25.831622 25316 leader_election.cc:288] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [CANDIDATE]: Term 1 election: Requested vote from peers 87877ea3035848769a83b861efb845e7 (127.17.31.2:43243), 4231fdc0f70f4ee083331c18cdeeccc0 (127.17.31.1:33945)
I20260807 03:09:25.832736 24914 raft_consensus.cc:2454] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7710219f3ce64aa48489cf3d46c4e070 in term 1.
I20260807 03:09:25.833532 25152 leader_election.cc:302] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4231fdc0f70f4ee083331c18cdeeccc0, 7710219f3ce64aa48489cf3d46c4e070; no voters: 87877ea3035848769a83b861efb845e7
I20260807 03:09:25.833535 24998 leader_election.cc:302] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87877ea3035848769a83b861efb845e7; no voters: 4231fdc0f70f4ee083331c18cdeeccc0, 7710219f3ce64aa48489cf3d46c4e070
I20260807 03:09:25.834080 25316 raft_consensus.cc:2781] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:25.834321 25315 raft_consensus.cc:2726] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 03:09:25.835495 25316 raft_consensus.cc:686] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [term 1 LEADER]: Becoming Leader. State: Replica: 7710219f3ce64aa48489cf3d46c4e070, State: Running, Role: LEADER
I20260807 03:09:25.836330 25316 consensus_queue.cc:237] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } } peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } }
I20260807 03:09:25.843745 23931 catalog_manager.cc:5495] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7710219f3ce64aa48489cf3d46c4e070 (127.17.31.3). New cstate: current_term: 1 leader_uuid: "7710219f3ce64aa48489cf3d46c4e070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } health_report { overall_health: HEALTHY } } }
W20260807 03:09:25.857174 25124 tablet.cc:2369] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:09:25.906172 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.901079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.17.31.3:38523)
0807 03:09:25.901306 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:25.901313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:25.901483 (+ 170us) server_negotiation.cc:407] Connection header received
0807 03:09:25.901637 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:25.901640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:25.901714 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:25.901834 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:09:25.902150 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.902936 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.904214 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.904611 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.904697 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.905108 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:09:25.905598 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:09:25.905710 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:25.905785 (+ 75us) server_negotiation.cc:299] Negotiation successful
0807 03:09:25.905848 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 03:09:25.914489 24914 raft_consensus.cc:1261] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0 [term 1 FOLLOWER]: Refusing update from remote peer 7710219f3ce64aa48489cf3d46c4e070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:09:25.914682 25069 raft_consensus.cc:1261] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [term 1 FOLLOWER]: Refusing update from remote peer 7710219f3ce64aa48489cf3d46c4e070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:09:25.915464 25321 consensus_queue.cc:1035] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:25.915936 25316 consensus_queue.cc:1035] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:25.954857 23929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:09:25.955310 23929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:09:25.964781 23929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a09cb5f898b4f0eaef384701af8613e: Sending DeleteTablet for 3 replicas of tablet 5e795ba096914c3897867a4160bd2349
I20260807 03:09:25.966135 25048 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e795ba096914c3897867a4160bd2349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:25 UTC) from {username='oryx', principal='oryx/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:41084
I20260807 03:09:25.966921 24893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e795ba096914c3897867a4160bd2349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:25 UTC) from {username='oryx', principal='oryx/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:57618
I20260807 03:09:25.967490 25201 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e795ba096914c3897867a4160bd2349 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:25 UTC) from {username='oryx', principal='oryx/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:42556
I20260807 03:09:25.972878 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 23898
I20260807 03:09:25.974215 23963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:25.977669 25337 tablet_replica.cc:330] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7: stopping tablet replica
I20260807 03:09:25.978245 25337 raft_consensus.cc:2229] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:25.979053 25337 raft_consensus.cc:2258] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:25.984316 25337 ts_tablet_manager.cc:1905] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:25.987875 25338 tablet_replica.cc:330] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0: stopping tablet replica
I20260807 03:09:25.988338 25338 raft_consensus.cc:2229] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:25.988945 25338 raft_consensus.cc:2258] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:25.990491 25339 tablet_replica.cc:330] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070: stopping tablet replica
I20260807 03:09:25.991094 25339 raft_consensus.cc:2229] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:09:25.994371 25338 ts_tablet_manager.cc:1905] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:25.995246 25337 ts_tablet_manager.cc:1918] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:09:25.995492 25337 log.cc:1196] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/5e795ba096914c3897867a4160bd2349
I20260807 03:09:25.996397 25337 ts_tablet_manager.cc:1939] T 5e795ba096914c3897867a4160bd2349 P 87877ea3035848769a83b861efb845e7: Deleting consensus metadata
I20260807 03:09:25.998584 25339 raft_consensus.cc:2258] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:26.000906 25339 ts_tablet_manager.cc:1905] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:26.005017 25338 ts_tablet_manager.cc:1918] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:09:26.005251 25338 log.cc:1196] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/5e795ba096914c3897867a4160bd2349
I20260807 03:09:26.006441 25338 ts_tablet_manager.cc:1939] T 5e795ba096914c3897867a4160bd2349 P 4231fdc0f70f4ee083331c18cdeeccc0: Deleting consensus metadata
I20260807 03:09:26.007684 25281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.969497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47394 (local address 127.17.31.61:37419)
0807 03:09:25.969720 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:25.969729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:25.971828 (+ 2099us) server_negotiation.cc:407] Connection header received
0807 03:09:25.973888 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:25.973894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:25.973986 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:25.974060 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:25.977948 (+ 3888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.980488 (+ 2540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.982868 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:25.983733 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:25.983835 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:25.989166 (+ 5331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:25.989174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:25.989179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:25.989234 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:25.995069 (+ 5835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:25.996558 (+ 1489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:25.996566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:25.996573 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:25.996788 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:25.997424 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:25.997429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:25.997432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:25.997950 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:25.998229 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:26.006844 (+ 8615us) server_negotiation.cc:299] Negotiation successful
0807 03:09:26.007079 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 03:09:26.013509 25339 ts_tablet_manager.cc:1918] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:09:26.013773 25339 log.cc:1196] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/5e795ba096914c3897867a4160bd2349
I20260807 03:09:26.014572 25339 ts_tablet_manager.cc:1939] T 5e795ba096914c3897867a4160bd2349 P 7710219f3ce64aa48489cf3d46c4e070: Deleting consensus metadata
I20260807 03:09:26.029832 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:25.972325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52206 (local address 127.17.31.60:41139)
0807 03:09:25.973509 (+ 1184us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:25.973515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:26.008425 (+ 34910us) server_negotiation.cc:407] Connection header received
0807 03:09:26.008510 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:26.008515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:26.008627 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:26.008729 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:26.011092 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:26.012699 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:26.015241 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:26.016092 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:26.016217 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:26.020924 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:26.020930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:26.020935 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:26.020992 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:26.025726 (+ 4734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:26.026764 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:26.026778 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:26.026782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:26.026970 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:26.027493 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:26.027498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:26.027502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:26.027983 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:26.028596 (+ 613us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:26.028859 (+ 263us) server_negotiation.cc:299] Negotiation successful
0807 03:09:26.029218 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058}
I20260807 03:09:26.136250 23918 catalog_manager.cc:4841] TS 7710219f3ce64aa48489cf3d46c4e070 (127.17.31.3:38523): tablet 5e795ba096914c3897867a4160bd2349 (table test-table [id=d5ecc0ef96ee4de5b9746487595e76f4]) successfully deleted
I20260807 03:09:26.136592 23915 catalog_manager.cc:4841] TS 4231fdc0f70f4ee083331c18cdeeccc0 (127.17.31.1:33945): tablet 5e795ba096914c3897867a4160bd2349 (table test-table [id=d5ecc0ef96ee4de5b9746487595e76f4]) successfully deleted
I20260807 03:09:26.136914 23915 catalog_manager.cc:4841] TS 87877ea3035848769a83b861efb845e7 (127.17.31.2:43243): tablet 5e795ba096914c3897867a4160bd2349 (table test-table [id=d5ecc0ef96ee4de5b9746487595e76f4]) successfully deleted
I20260807 03:09:26.137504 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 23898
I20260807 03:09:26.195477 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:26.166928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.17.31.60:41139)
0807 03:09:26.167149 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:26.167155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:26.178852 (+ 11697us) server_negotiation.cc:407] Connection header received
0807 03:09:26.179130 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:26.179137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:26.179228 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:26.179302 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:26.181638 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:26.182798 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:26.185158 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:26.185871 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:26.186000 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:26.189171 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:26.189178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:26.189182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:26.189222 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:26.192702 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:26.193483 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:26.193493 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:26.193495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:26.193609 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:26.193983 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:26.193989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:26.193992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:26.194417 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:26.194696 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:26.195000 (+ 304us) server_negotiation.cc:299] Negotiation successful
0807 03:09:26.195198 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:09:26.198633 25281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:26.166573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47400 (local address 127.17.31.61:37419)
0807 03:09:26.166775 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:26.166781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:26.181977 (+ 15196us) server_negotiation.cc:407] Connection header received
0807 03:09:26.182184 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:26.182193 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:26.182331 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:26.182444 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:26.184708 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:26.185650 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:26.188167 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:26.188840 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:26.188977 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:26.192171 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:26.192176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:26.192180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:26.192216 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:26.195718 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:26.196664 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:26.196669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:26.196673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:26.196817 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:26.197180 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:26.197185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:26.197189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:26.197552 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:26.197774 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:26.198032 (+ 258us) server_negotiation.cc:299] Negotiation successful
0807 03:09:26.198248 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 03:09:26.949936 25350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:26.949027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.1:37769)
0807 03:09:26.949597 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:26.949751 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":266,"threads_started":1}
W20260807 03:09:26.950513 24962 heartbeater.cc:644] Failed to heartbeat to 127.17.31.62:44907 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
I20260807 03:09:26.950719 25351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:26.949902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.2:50837)
0807 03:09:26.950390 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:26.950539 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":388,"threads_started":1}
W20260807 03:09:26.951104 25122 heartbeater.cc:644] Failed to heartbeat to 127.17.31.62:44907 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
I20260807 03:09:26.951807 25352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:26.951112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.3:35611)
0807 03:09:26.951548 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:26.951656 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":247,"threads_started":1}
W20260807 03:09:26.952142 25270 heartbeater.cc:644] Failed to heartbeat to 127.17.31.62:44907 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
I20260807 03:09:27.463660 25353 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6a09cb5f898b4f0eaef384701af8613e)
I20260807 03:09:27.463995 25353 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } }
I20260807 03:09:27.465641 25353 leader_election.cc:288] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a09cb5f898b4f0eaef384701af8613e (127.17.31.62:44907), 9bfc45b1240243d4b9701258adfb2723 (127.17.31.60:41139)
I20260807 03:09:27.465691 25354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6a09cb5f898b4f0eaef384701af8613e)
I20260807 03:09:27.465994 25354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } }
I20260807 03:09:27.467005 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.465761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.0.0.1:50032)
0807 03:09:27.466418 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.466661 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":858,"threads_started":1}
W20260807 03:09:27.467419 24551 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
I20260807 03:09:27.475847 25360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.467690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.0.0.1:50040)
0807 03:09:27.475175 (+ 7485us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.475418 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7321,"threads_started":1}
W20260807 03:09:27.476655 24754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
I20260807 03:09:27.477054 25356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.476405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.0.0.1:50052)
0807 03:09:27.476568 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.476805 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260807 03:09:27.477216 24551 leader_election.cc:334] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6a09cb5f898b4f0eaef384701af8613e (127.17.31.62:44907): Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
I20260807 03:09:27.478434 25354 leader_election.cc:288] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6a09cb5f898b4f0eaef384701af8613e (127.17.31.62:44907), c8706df62d4147e7a32dc4b6e6d2511a (127.17.31.61:37419)
I20260807 03:09:27.480082 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.479350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.0.0.1:50056)
0807 03:09:27.479659 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.479835 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
W20260807 03:09:27.480471 24754 leader_election.cc:334] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6a09cb5f898b4f0eaef384701af8613e (127.17.31.62:44907): Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
I20260807 03:09:27.481372 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.466068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:41139 (local address 127.0.0.1:52226)
0807 03:09:27.472655 (+ 6587us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.472681 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:27.472956 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:27.473716 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:27.473723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:27.473754 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:27.474086 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:27.474104 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.476662 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.476668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:27.478760 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:27.478777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.480625 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.480631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:27.480806 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:27.480829 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 03:09:27.480944 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 03:09:27.481079 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6457,"thread_start_us":9265,"threads_started":1}
I20260807 03:09:27.481482 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.466486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.17.31.60:41139)
0807 03:09:27.473352 (+ 6866us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:27.473361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:27.473386 (+ 25us) server_negotiation.cc:407] Connection header received
0807 03:09:27.473471 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:27.473479 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:27.473562 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:27.473668 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:27.474323 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.475877 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.478966 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.480444 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.480610 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:27.480704 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:27.480968 (+ 264us) server_negotiation.cc:299] Negotiation successful
0807 03:09:27.481092 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":6704,"thread_start_us":7342,"threads_started":1}
I20260807 03:09:27.482057 24782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9bfc45b1240243d4b9701258adfb2723" is_pre_election: true
I20260807 03:09:27.482402 24782 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8706df62d4147e7a32dc4b6e6d2511a in term 1.
I20260807 03:09:27.482903 24552 leader_election.cc:302] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfc45b1240243d4b9701258adfb2723, c8706df62d4147e7a32dc4b6e6d2511a; no voters: 6a09cb5f898b4f0eaef384701af8613e
I20260807 03:09:27.483304 25353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 03:09:27.483444 25353 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6a09cb5f898b4f0eaef384701af8613e)
I20260807 03:09:27.483573 25353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:09:27.483959 25362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.475549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.17.31.61:37419)
0807 03:09:27.476830 (+ 1281us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:27.476836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:27.476858 (+ 22us) server_negotiation.cc:407] Connection header received
0807 03:09:27.476924 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:27.476935 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:27.477018 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:27.477836 (+ 818us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:27.478152 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.479717 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.481747 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.482996 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.483136 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:27.483241 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:27.483568 (+ 327us) server_negotiation.cc:299] Negotiation successful
0807 03:09:27.483699 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":1883,"threads_started":1}
I20260807 03:09:27.484012 25360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.473865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:37419 (local address 127.0.0.1:47414)
0807 03:09:27.476069 (+ 2204us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.476092 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:27.476224 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:27.477211 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:27.477219 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:27.477253 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:27.477675 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:27.477690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.479861 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.479867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:27.481546 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:27.481563 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.483207 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.483213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:27.483401 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:27.483444 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 03:09:27.483542 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 03:09:27.483683 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2097,"mutex_wait_us":42,"thread_start_us":5625,"threads_started":1}
I20260807 03:09:27.484715 24579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9bfc45b1240243d4b9701258adfb2723" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" is_pre_election: true
I20260807 03:09:27.485909 25353 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } }
I20260807 03:09:27.486369 24579 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfc45b1240243d4b9701258adfb2723 in current term 2: Already voted for candidate c8706df62d4147e7a32dc4b6e6d2511a in this term.
I20260807 03:09:27.486683 25353 leader_election.cc:288] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [CANDIDATE]: Term 2 election: Requested vote from peers 6a09cb5f898b4f0eaef384701af8613e (127.17.31.62:44907), 9bfc45b1240243d4b9701258adfb2723 (127.17.31.60:41139)
I20260807 03:09:27.487042 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.486669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.0.0.1:50068)
0807 03:09:27.486867 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.486932 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260807 03:09:27.487222 24782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9bfc45b1240243d4b9701258adfb2723"
I20260807 03:09:27.487546 24782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:09:27.487654 24756 leader_election.cc:302] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfc45b1240243d4b9701258adfb2723; no voters: 6a09cb5f898b4f0eaef384701af8613e, c8706df62d4147e7a32dc4b6e6d2511a
I20260807 03:09:27.488023 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.487602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.0.0.1:50072)
0807 03:09:27.487791 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.487890 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260807 03:09:27.488161 24551 leader_election.cc:334] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6a09cb5f898b4f0eaef384701af8613e (127.17.31.62:44907): Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
I20260807 03:09:27.489833 24782 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8706df62d4147e7a32dc4b6e6d2511a in term 2.
I20260807 03:09:27.490006 25354 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:09:27.490284 24552 leader_election.cc:302] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfc45b1240243d4b9701258adfb2723, c8706df62d4147e7a32dc4b6e6d2511a; no voters: 6a09cb5f898b4f0eaef384701af8613e
I20260807 03:09:27.490638 25353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:09:27.491014 25353 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [term 2 LEADER]: Becoming Leader. State: Replica: c8706df62d4147e7a32dc4b6e6d2511a, State: Running, Role: LEADER
I20260807 03:09:27.491394 25353 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } }
I20260807 03:09:27.493253 25364 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8706df62d4147e7a32dc4b6e6d2511a. Latest consensus state: current_term: 2 leader_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } } }
I20260807 03:09:27.493703 25364 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:27.494377 25367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:09:27.498081 25367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:09:27.498943 25367 catalog_manager.cc:1259] Loaded cluster ID: a5db12aebde34806b4cc58fef502962e
I20260807 03:09:27.499038 25367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:09:27.500157 25367 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:09:27.500993 25367 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: Loaded TSK: 0
I20260807 03:09:27.501827 25367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:09:27.523530 24568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:09:27.530253 24568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:09:27.534420 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.533948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.0.0.1:50096)
0807 03:09:27.534157 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.534229 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 03:09:27.534459 24782 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [term 2 FOLLOWER]: Refusing update from remote peer c8706df62d4147e7a32dc4b6e6d2511a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260807 03:09:27.535262 25364 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260807 03:09:27.535468 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.535094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.0.0.1:50112)
0807 03:09:27.535268 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.535346 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260807 03:09:27.535619 24551 consensus_peers.cc:487] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a -> Peer 6a09cb5f898b4f0eaef384701af8613e (127.17.31.62:44907): Couldn't send request to peer 6a09cb5f898b4f0eaef384701af8613e. Status: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 03:09:27.537930 25354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8706df62d4147e7a32dc4b6e6d2511a. Latest consensus state: current_term: 2 leader_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } } }
I20260807 03:09:27.538261 25354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:09:27.539671 25353 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } } }
I20260807 03:09:27.540158 25353 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:27.540454 25369 mvcc.cc:205] Tried to move back new op lower bound from 7315751598214504448 to 7315751598048219136. Current Snapshot: MvccSnapshot[applied={T|T < 7315751598214504448}]
I20260807 03:09:27.540907 25354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6a09cb5f898b4f0eaef384701af8613e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44907 } } peers { permanent_uuid: "c8706df62d4147e7a32dc4b6e6d2511a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 37419 } attrs { promote: false } } peers { permanent_uuid: "9bfc45b1240243d4b9701258adfb2723" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 41139 } attrs { promote: false } } }
I20260807 03:09:27.541177 25354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bfc45b1240243d4b9701258adfb2723 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:09:27.558353 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.548057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42568 (local address 127.17.31.3:38523)
0807 03:09:27.548556 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:27.548563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:27.548584 (+ 21us) server_negotiation.cc:407] Connection header received
0807 03:09:27.548658 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:27.548662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:27.548745 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:27.549526 (+ 781us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:27.549774 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.551062 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.556006 (+ 4944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.557130 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.557265 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:27.557370 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:27.557844 (+ 474us) server_negotiation.cc:299] Negotiation successful
0807 03:09:27.558001 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":681,"threads_started":1}
I20260807 03:09:27.558442 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.547208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:33945 (local address 127.0.0.1:57624)
0807 03:09:27.547381 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.547400 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:27.547511 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:27.548824 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:27.548831 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:27.548864 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:27.549155 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:27.549170 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.551266 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.551278 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:27.556149 (+ 4871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:27.556168 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.557667 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.557673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:27.557868 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:27.557903 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 03:09:27.557989 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 03:09:27.558122 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":2805}
I20260807 03:09:27.558682 25356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.547685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:38523 (local address 127.0.0.1:42568)
0807 03:09:27.547844 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.547861 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:27.547966 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:27.548901 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:27.548906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:27.548932 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:27.549353 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:27.549368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.551215 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.551220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:27.553942 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:27.553963 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.557361 (+ 3398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.557369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:27.557616 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:27.557658 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 03:09:27.557758 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 03:09:27.557985 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":57}
I20260807 03:09:27.559506 25374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.547652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57624 (local address 127.17.31.1:33945)
0807 03:09:27.548462 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:27.548468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:27.548487 (+ 19us) server_negotiation.cc:407] Connection header received
0807 03:09:27.548566 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:27.548574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:27.548661 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:27.549334 (+ 673us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:27.549572 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.551040 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.556333 (+ 5293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.557505 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.557653 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:27.557745 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:27.559101 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0807 03:09:27.559247 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":977,"threads_started":1}
I20260807 03:09:27.559484 25201 tablet_service.cc:1467] Processing CreateTablet for tablet 34d83076ddb240fc89da82de8893fe0f (DEFAULT_TABLE table=test-table [id=0357550b5acf4d31bd24df37bcc5c04e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:27.559999 25201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d83076ddb240fc89da82de8893fe0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:27.560441 24893 tablet_service.cc:1467] Processing CreateTablet for tablet 34d83076ddb240fc89da82de8893fe0f (DEFAULT_TABLE table=test-table [id=0357550b5acf4d31bd24df37bcc5c04e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:27.560858 24893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d83076ddb240fc89da82de8893fe0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:27.562870 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.548874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.17.31.2:43243)
0807 03:09:27.555613 (+ 6739us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:27.555618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:27.555631 (+ 13us) server_negotiation.cc:407] Connection header received
0807 03:09:27.555687 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:27.555690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:27.555742 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:27.555820 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:27.556495 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.557160 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.559719 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.561744 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.561886 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:27.561997 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:27.562400 (+ 403us) server_negotiation.cc:299] Negotiation successful
0807 03:09:27.562546 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6648,"thread_start_us":6565,"threads_started":1}
I20260807 03:09:27.562919 25376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.548475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:43243 (local address 127.0.0.1:41090)
0807 03:09:27.555164 (+ 6689us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.555185 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:27.555284 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:27.555869 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:27.555874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:27.555905 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:27.556348 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:27.556358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.557339 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.557346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:27.559498 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:27.559514 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.561947 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.561953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:27.562129 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:27.562151 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 03:09:27.562328 (+ 177us) client_negotiation.cc:240] Negotiation successful
0807 03:09:27.562498 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6560,"mutex_wait_us":1,"thread_start_us":6035,"threads_started":1}
I20260807 03:09:27.564019 25048 tablet_service.cc:1467] Processing CreateTablet for tablet 34d83076ddb240fc89da82de8893fe0f (DEFAULT_TABLE table=test-table [id=0357550b5acf4d31bd24df37bcc5c04e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:27.564463 25048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d83076ddb240fc89da82de8893fe0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:27.574259 25381 tablet_bootstrap.cc:492] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7: Bootstrap starting.
I20260807 03:09:27.574930 25380 tablet_bootstrap.cc:492] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070: Bootstrap starting.
I20260807 03:09:27.575189 25379 tablet_bootstrap.cc:492] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0: Bootstrap starting.
I20260807 03:09:27.576084 25381 tablet_bootstrap.cc:654] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:27.576596 25379 tablet_bootstrap.cc:654] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:27.576762 25380 tablet_bootstrap.cc:654] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:27.579627 25379 tablet_bootstrap.cc:492] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0: No bootstrap required, opened a new log
I20260807 03:09:27.579793 25379 ts_tablet_manager.cc:1397] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260807 03:09:27.579876 25381 tablet_bootstrap.cc:492] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7: No bootstrap required, opened a new log
I20260807 03:09:27.580032 25381 ts_tablet_manager.cc:1397] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260807 03:09:27.580411 25380 tablet_bootstrap.cc:492] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070: No bootstrap required, opened a new log
I20260807 03:09:27.580562 25380 ts_tablet_manager.cc:1397] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260807 03:09:27.580641 25379 raft_consensus.cc:348] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } } peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } }
I20260807 03:09:27.580890 25379 raft_consensus.cc:374] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:27.580803 25381 raft_consensus.cc:348] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } } peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } }
I20260807 03:09:27.580993 25379 raft_consensus.cc:729] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4231fdc0f70f4ee083331c18cdeeccc0, State: Initialized, Role: FOLLOWER
I20260807 03:09:27.581043 25381 raft_consensus.cc:374] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:27.581147 25381 raft_consensus.cc:729] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87877ea3035848769a83b861efb845e7, State: Initialized, Role: FOLLOWER
I20260807 03:09:27.581269 25379 consensus_queue.cc:260] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } } peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } }
I20260807 03:09:27.581380 25380 raft_consensus.cc:348] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } } peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } }
I20260807 03:09:27.581393 25381 consensus_queue.cc:260] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } } peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } }
I20260807 03:09:27.581609 25380 raft_consensus.cc:374] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:27.581740 25380 raft_consensus.cc:729] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7710219f3ce64aa48489cf3d46c4e070, State: Initialized, Role: FOLLOWER
I20260807 03:09:27.582036 25380 consensus_queue.cc:260] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } } peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } }
I20260807 03:09:27.584746 25380 ts_tablet_manager.cc:1428] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260807 03:09:27.592473 25271 heartbeater.cc:503] Master 127.17.31.61:37419 was elected leader, sending a full tablet report...
I20260807 03:09:27.596630 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.585413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.3:53189)
0807 03:09:27.596111 (+ 10698us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.596279 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10529,"threads_started":1}
I20260807 03:09:27.598049 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.597527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.3:56791)
0807 03:09:27.597748 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.597872 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 03:09:27.598217 25379 ts_tablet_manager.cc:1428] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0: Time spent starting tablet: real 0.018s user 0.000s sys 0.019s
I20260807 03:09:27.598505 25381 ts_tablet_manager.cc:1428] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7: Time spent starting tablet: real 0.018s user 0.005s sys 0.012s
I20260807 03:09:27.602488 24963 heartbeater.cc:503] Master 127.17.31.61:37419 was elected leader, sending a full tablet report...
I20260807 03:09:27.603276 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.598914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.1:42487)
0807 03:09:27.602882 (+ 3968us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.603017 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3841,"threads_started":1}
I20260807 03:09:27.604140 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.603194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.2:35463)
0807 03:09:27.603758 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.603893 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":334,"threads_started":1}
I20260807 03:09:27.605031 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.604658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.2:53299)
0807 03:09:27.604803 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.604891 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 03:09:27.605031 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.604364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.1:50903)
0807 03:09:27.604820 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.604894 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260807 03:09:27.605336 25123 heartbeater.cc:503] Master 127.17.31.61:37419 was elected leader, sending a full tablet report...
I20260807 03:09:27.721962 25385 raft_consensus.cc:482] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:09:27.722177 25385 raft_consensus.cc:504] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } } peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } }
I20260807 03:09:27.723922 25385 leader_election.cc:288] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7710219f3ce64aa48489cf3d46c4e070 (127.17.31.3:38523), 87877ea3035848769a83b861efb845e7 (127.17.31.2:43243)
I20260807 03:09:27.731387 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.724172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:38523 (local address 127.17.31.1:58779)
0807 03:09:27.724446 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.724466 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:27.724612 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:27.725034 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:27.725041 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:27.725094 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:27.725426 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:27.725439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.726728 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.726731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:27.728968 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:27.728984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.730630 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.730637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:27.730806 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:27.730856 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 03:09:27.730956 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 03:09:27.731106 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":105}
I20260807 03:09:27.731429 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.724256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:58779 (local address 127.17.31.3:38523)
0807 03:09:27.724458 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:27.724465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:27.724569 (+ 104us) server_negotiation.cc:407] Connection header received
0807 03:09:27.724755 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:27.724763 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:27.724882 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:27.724986 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:27.725564 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.726613 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.729136 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.730426 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.730559 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:27.730731 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:27.731009 (+ 278us) server_negotiation.cc:299] Negotiation successful
0807 03:09:27.731141 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 03:09:27.731680 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.724322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:44345 (local address 127.17.31.2:43243)
0807 03:09:27.724507 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:27.724511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:27.725285 (+ 774us) server_negotiation.cc:407] Connection header received
0807 03:09:27.725455 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:27.725459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:27.725547 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:27.725644 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:27.726067 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.726995 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.729284 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.730573 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.730677 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:27.730777 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:27.731288 (+ 511us) server_negotiation.cc:299] Negotiation successful
0807 03:09:27.731436 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 03:09:27.732340 25222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d83076ddb240fc89da82de8893fe0f" candidate_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7710219f3ce64aa48489cf3d46c4e070" is_pre_election: true
I20260807 03:09:27.732482 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.724311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:43243 (local address 127.17.31.1:44345)
0807 03:09:27.725168 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.725187 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:27.725328 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:27.725660 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:27.725663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:27.725683 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:27.725923 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:27.725930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.727116 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.727122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:27.729141 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:27.729158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.730890 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.730894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:27.731103 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:27.731134 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 03:09:27.731960 (+ 826us) client_negotiation.cc:240] Negotiation successful
0807 03:09:27.732094 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":34,"thread_start_us":321,"threads_started":1}
I20260807 03:09:27.732620 25222 raft_consensus.cc:2454] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4231fdc0f70f4ee083331c18cdeeccc0 in term 0.
I20260807 03:09:27.733115 25069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d83076ddb240fc89da82de8893fe0f" candidate_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87877ea3035848769a83b861efb845e7" is_pre_election: true
I20260807 03:09:27.733151 24847 leader_election.cc:302] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4231fdc0f70f4ee083331c18cdeeccc0, 7710219f3ce64aa48489cf3d46c4e070; no voters:
I20260807 03:09:27.733359 25069 raft_consensus.cc:2454] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4231fdc0f70f4ee083331c18cdeeccc0 in term 0.
I20260807 03:09:27.733561 25385 raft_consensus.cc:2781] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:09:27.733680 25385 raft_consensus.cc:482] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:09:27.733808 25385 raft_consensus.cc:3037] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:27.736861 25385 raft_consensus.cc:504] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } } peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } }
I20260807 03:09:27.737546 25385 leader_election.cc:288] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [CANDIDATE]: Term 1 election: Requested vote from peers 7710219f3ce64aa48489cf3d46c4e070 (127.17.31.3:38523), 87877ea3035848769a83b861efb845e7 (127.17.31.2:43243)
I20260807 03:09:27.737967 25222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d83076ddb240fc89da82de8893fe0f" candidate_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7710219f3ce64aa48489cf3d46c4e070"
I20260807 03:09:27.737964 25069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d83076ddb240fc89da82de8893fe0f" candidate_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87877ea3035848769a83b861efb845e7"
I20260807 03:09:27.738168 25069 raft_consensus.cc:3037] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:27.738181 25222 raft_consensus.cc:3037] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:27.740844 25222 raft_consensus.cc:2454] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4231fdc0f70f4ee083331c18cdeeccc0 in term 1.
I20260807 03:09:27.741065 25069 raft_consensus.cc:2454] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4231fdc0f70f4ee083331c18cdeeccc0 in term 1.
I20260807 03:09:27.741338 24847 leader_election.cc:302] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4231fdc0f70f4ee083331c18cdeeccc0, 7710219f3ce64aa48489cf3d46c4e070; no voters:
I20260807 03:09:27.741649 25385 raft_consensus.cc:2781] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:27.742130 25385 raft_consensus.cc:686] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [term 1 LEADER]: Becoming Leader. State: Replica: 4231fdc0f70f4ee083331c18cdeeccc0, State: Running, Role: LEADER
I20260807 03:09:27.742554 25385 consensus_queue.cc:237] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } } peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } }
I20260807 03:09:27.745035 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.743123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.1:58471)
0807 03:09:27.744668 (+ 1545us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.744793 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1312}
I20260807 03:09:27.748930 24567 catalog_manager.cc:5495] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4231fdc0f70f4ee083331c18cdeeccc0 (127.17.31.1). New cstate: current_term: 1 leader_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4231fdc0f70f4ee083331c18cdeeccc0" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 33945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 } health_report { overall_health: UNKNOWN } } }
I20260807 03:09:27.792956 25374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.787856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57626 (local address 127.17.31.1:33945)
0807 03:09:27.788045 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:27.788049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:27.788067 (+ 18us) server_negotiation.cc:407] Connection header received
0807 03:09:27.788241 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:27.788245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:27.788333 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:27.788455 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:09:27.788823 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.789533 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.790895 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.791348 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.791444 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:27.791883 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:09:27.792461 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:09:27.792581 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:27.792698 (+ 117us) server_negotiation.cc:299] Negotiation successful
0807 03:09:27.792761 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 03:09:27.796936 25069 raft_consensus.cc:1261] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7 [term 1 FOLLOWER]: Refusing update from remote peer 4231fdc0f70f4ee083331c18cdeeccc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:09:27.797184 25222 raft_consensus.cc:1261] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070 [term 1 FOLLOWER]: Refusing update from remote peer 4231fdc0f70f4ee083331c18cdeeccc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:09:27.797755 25392 consensus_queue.cc:1035] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87877ea3035848769a83b861efb845e7" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:27.798108 25385 consensus_queue.cc:1035] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7710219f3ce64aa48489cf3d46c4e070" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 38523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:27.800931 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.800352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.3:42797)
0807 03:09:27.800605 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.800712 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260807 03:09:27.800976 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.800355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.2:60571)
0807 03:09:27.800574 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.800743 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260807 03:09:27.806202 25396 mvcc.cc:205] Tried to move back new op lower bound from 7315751599289323520 to 7315751599078211584. Current Snapshot: MvccSnapshot[applied={T|T < 7315751599289323520}]
I20260807 03:09:27.807407 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.806968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.3:33717)
0807 03:09:27.807168 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.807254 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260807 03:09:27.808208 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.807501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.2:42511)
0807 03:09:27.807852 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.807966 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222}
I20260807 03:09:27.815243 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.814703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.1:52537)
0807 03:09:27.814941 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.815058 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260807 03:09:27.819252 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.818840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.17.31.1:45383)
0807 03:09:27.818996 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.819081 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 03:09:27.825035 24567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:09:27.825316 24567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:09:27.830953 24567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8706df62d4147e7a32dc4b6e6d2511a: Sending DeleteTablet for 3 replicas of tablet 34d83076ddb240fc89da82de8893fe0f
I20260807 03:09:27.831950 25201 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d83076ddb240fc89da82de8893fe0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:27 UTC) from {username='oryx', principal='oryx/127.17.31.61@KRBTEST.COM'} at 127.0.0.1:42568
I20260807 03:09:27.831966 24893 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d83076ddb240fc89da82de8893fe0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:27 UTC) from {username='oryx', principal='oryx/127.17.31.61@KRBTEST.COM'} at 127.0.0.1:57624
I20260807 03:09:27.835995 25048 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d83076ddb240fc89da82de8893fe0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:27 UTC) from {username='oryx', principal='oryx/127.17.31.61@KRBTEST.COM'} at 127.0.0.1:41090
I20260807 03:09:27.842767 25410 tablet_replica.cc:330] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7: stopping tablet replica
I20260807 03:09:27.842989 25410 raft_consensus.cc:2229] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:27.843317 25410 raft_consensus.cc:2258] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:27.849721 25407 tablet_replica.cc:330] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0: stopping tablet replica
I20260807 03:09:27.850056 25407 raft_consensus.cc:2229] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:09:27.851020 25407 raft_consensus.cc:2258] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:27.851615 25408 tablet_replica.cc:330] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070: stopping tablet replica
I20260807 03:09:27.852555 25407 ts_tablet_manager.cc:1905] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:27.852813 25408 raft_consensus.cc:2229] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:27.854055 25408 raft_consensus.cc:2258] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:27.855181 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 24758
I20260807 03:09:27.855347 25410 ts_tablet_manager.cc:1905] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:27.861409 25408 ts_tablet_manager.cc:1905] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:27.862106 25410 ts_tablet_manager.cc:1918] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:09:27.862246 25410 log.cc:1196] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/34d83076ddb240fc89da82de8893fe0f
I20260807 03:09:27.862802 25410 ts_tablet_manager.cc:1939] T 34d83076ddb240fc89da82de8893fe0f P 87877ea3035848769a83b861efb845e7: Deleting consensus metadata
I20260807 03:09:27.863963 24551 catalog_manager.cc:4841] TS 87877ea3035848769a83b861efb845e7 (127.17.31.2:43243): tablet 34d83076ddb240fc89da82de8893fe0f (table test-table [id=0357550b5acf4d31bd24df37bcc5c04e]) successfully deleted
I20260807 03:09:27.865615 25407 ts_tablet_manager.cc:1918] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:09:27.865846 25407 log.cc:1196] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/34d83076ddb240fc89da82de8893fe0f
I20260807 03:09:27.866451 25407 ts_tablet_manager.cc:1939] T 34d83076ddb240fc89da82de8893fe0f P 4231fdc0f70f4ee083331c18cdeeccc0: Deleting consensus metadata
I20260807 03:09:27.868345 25408 ts_tablet_manager.cc:1918] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:09:27.868356 24551 catalog_manager.cc:4841] TS 4231fdc0f70f4ee083331c18cdeeccc0 (127.17.31.1:33945): tablet 34d83076ddb240fc89da82de8893fe0f (table test-table [id=0357550b5acf4d31bd24df37bcc5c04e]) successfully deleted
I20260807 03:09:27.868474 25408 log.cc:1196] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/34d83076ddb240fc89da82de8893fe0f
I20260807 03:09:27.868912 25408 ts_tablet_manager.cc:1939] T 34d83076ddb240fc89da82de8893fe0f P 7710219f3ce64aa48489cf3d46c4e070: Deleting consensus metadata
I20260807 03:09:27.870055 24554 catalog_manager.cc:4841] TS 7710219f3ce64aa48489cf3d46c4e070 (127.17.31.3:38523): tablet 34d83076ddb240fc89da82de8893fe0f (table test-table [id=0357550b5acf4d31bd24df37bcc5c04e]) successfully deleted
I20260807 03:09:27.885576 25374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.856324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.17.31.1:33945)
0807 03:09:27.856490 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:27.856495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:27.870346 (+ 13851us) server_negotiation.cc:407] Connection header received
0807 03:09:27.870567 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:27.870573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:27.870658 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:27.870759 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:27.872621 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.873804 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.875742 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:27.876184 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:27.876280 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:27.879579 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:27.879590 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:27.879605 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:27.879634 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:27.882830 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:27.883619 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:27.883626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:27.883629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:27.883776 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:27.884177 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:27.884180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:27.884182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:27.884498 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:27.884745 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:27.885073 (+ 328us) server_negotiation.cc:299] Negotiation successful
0807 03:09:27.885228 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 03:09:27.886242 24955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:27.984671 25376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.984086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.0.0.1:50116)
0807 03:09:27.984313 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.984474 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 03:09:27.985805 25376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:27.985457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44907 (local address 127.0.0.1:50122)
0807 03:09:27.985625 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:27.985700 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260807 03:09:27.993196 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 24758
I20260807 03:09:28.014076 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 24973
I20260807 03:09:28.027878 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:28.014915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.17.31.2:43243)
0807 03:09:28.015142 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:28.015148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:28.015248 (+ 100us) server_negotiation.cc:407] Connection header received
0807 03:09:28.015511 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:28.015515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:28.015591 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:28.015695 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:28.017006 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:28.017757 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:28.019239 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:28.019649 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:28.019800 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:28.022227 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:28.022243 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:28.022260 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:28.022295 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:28.025015 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:28.025706 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:28.025716 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:28.025719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:28.025860 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:28.026246 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:28.026252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:28.026255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:28.026690 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:28.027022 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:28.027308 (+ 286us) server_negotiation.cc:299] Negotiation successful
0807 03:09:28.027526 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 03:09:28.028527 25116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:28.125509 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 24973
I20260807 03:09:28.145044 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 25136
I20260807 03:09:28.158661 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:28.145804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.17.31.3:38523)
0807 03:09:28.146005 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:28.146012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:28.146081 (+ 69us) server_negotiation.cc:407] Connection header received
0807 03:09:28.146329 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:28.146335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:28.146415 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:28.146540 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:28.147851 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:28.148607 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:28.149807 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:28.150219 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:28.150374 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:28.152841 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:28.152865 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:28.152891 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:28.152921 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:28.155629 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:28.156279 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:28.156296 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:28.156299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:28.156430 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:28.156811 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:28.156819 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:28.156822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:28.157174 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:28.157480 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:28.157806 (+ 326us) server_negotiation.cc:299] Negotiation successful
0807 03:09:28.158035 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 03:09:28.159296 25264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:28.279003 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 25136
I20260807 03:09:28.301877 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 24424
I20260807 03:09:28.303067 24600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:28.432771 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 24424
W20260807 03:09:28.455144 24757 connection.cc:535] server connection from 127.0.0.1:52226 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 03:09:28.455560 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 24530
I20260807 03:09:28.456568 24802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:28.573999 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 24530
2026-08-07T03:09:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11396 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:09:28 dist-test-slave-tm1g krb5kdc[25420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:09:28 dist-test-slave-tm1g krb5kdc[25420](info): set up 2 sockets
Aug 07 03:09:28 dist-test-slave-tm1g krb5kdc[25420](info): commencing operation
krb5kdc: starting...
W20260807 03:09:29.684846 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:09:29 dist-test-slave-tm1g krb5kdc[25420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:09:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:09:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.62.keytab.
Entry for principal oryx/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.62.keytab.
I20260807 03:09:29.906181 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:41397
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.62.keytab
--principal=oryx/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:41397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:30.224527 25433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:30.224874 25433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:30.225040 25433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:30.232060 25433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:09:30.232199 25433 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 03:09:30.232322 25433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:30.232436 25433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:30.232538 25433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:09:30.232627 25433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:09:30.242094 25433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:41397
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.62.keytab
--principal=oryx/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:41397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:30.243330 25433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:30.244652 25433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:30.256407 25438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:30.256487 25439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:30.256445 25441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:30.257323 25433 server_base.cc:1034] running on GCE node
I20260807 03:09:30.258028 25433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:30.259358 25433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:30.260571 25433 hybrid_clock.cc:648] HybridClock initialized: now 1786072170260519 us; error 29 us; skew 500 ppm
I20260807 03:09:30.260887 25433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:30 dist-test-slave-tm1g krb5kdc[25420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072170, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:30.263935 25433 init.cc:377] Logged in from keytab as oryx/127.17.31.62@KRBTEST.COM (short username oryx)
I20260807 03:09:30.265286 25433 webserver.cc:460] Webserver started at http://127.17.31.62:39657/ using document root <none> and password file <none>
I20260807 03:09:30.265936 25433 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:30.266021 25433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:30.266281 25433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:30.269119 25433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "dd16109787144a5d8e7bb7550b8a70bb"
format_stamp: "Formatted at 2026-08-07 03:09:30 on dist-test-slave-tm1g"
server_key: "7be14c7bb213fa7ef35a3421be547e42"
server_key_iv: "2213c942bf545fab8324141101b53795"
server_key_version: "encryptionkey@0"
I20260807 03:09:30.269848 25433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "dd16109787144a5d8e7bb7550b8a70bb"
format_stamp: "Formatted at 2026-08-07 03:09:30 on dist-test-slave-tm1g"
server_key: "7be14c7bb213fa7ef35a3421be547e42"
server_key_iv: "2213c942bf545fab8324141101b53795"
server_key_version: "encryptionkey@0"
I20260807 03:09:30.274704 25433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 03:09:30.277834 25449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:30.279141 25433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:09:30.279273 25433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "dd16109787144a5d8e7bb7550b8a70bb"
format_stamp: "Formatted at 2026-08-07 03:09:30 on dist-test-slave-tm1g"
server_key: "7be14c7bb213fa7ef35a3421be547e42"
server_key_iv: "2213c942bf545fab8324141101b53795"
server_key_version: "encryptionkey@0"
I20260807 03:09:30.279436 25433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:30.291056 25433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:30.295171 25433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:30.295382 25433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:30.316435 25433 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:41397
I20260807 03:09:30.316468 25517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:41397 every 8 connection(s)
I20260807 03:09:30.317737 25433 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:09:30.321409 25518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:30.324971 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 25433
I20260807 03:09:30.325224 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:09:30.325853 17532 external_mini_cluster.cc:1428] Setting key 51cb66519839d054d9701e0b947e5468
I20260807 03:09:30.332113 25518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb: Bootstrap starting.
I20260807 03:09:30.335623 25518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:30.336733 25518 log.cc:824] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:30.339314 25518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb: No bootstrap required, opened a new log
Aug 07 03:09:30 dist-test-slave-tm1g krb5kdc[25420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM
I20260807 03:09:30.348562 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:30.327803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57862 (local address 127.17.31.62:41397)
0807 03:09:30.328573 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:30.328583 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:30.328627 (+ 44us) server_negotiation.cc:407] Connection header received
0807 03:09:30.330642 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:30.330663 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:30.331009 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:30.331296 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:30.333081 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:30.334431 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:30.336219 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:30.336944 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:30.337079 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:30.341512 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:30.341520 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:30.341533 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:30.341566 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:30.344635 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:30.345488 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:30.345494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:30.345499 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:30.345658 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:30.346062 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:30.346065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:30.346067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:30.346588 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:30.346812 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:30.347247 (+ 435us) server_negotiation.cc:299] Negotiation successful
0807 03:09:30.347577 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":376,"threads_started":1}
I20260807 03:09:30.348644 25518 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd16109787144a5d8e7bb7550b8a70bb" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41397 } }
I20260807 03:09:30.349099 25518 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:30.349221 25518 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd16109787144a5d8e7bb7550b8a70bb, State: Initialized, Role: FOLLOWER
I20260807 03:09:30.349750 25518 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd16109787144a5d8e7bb7550b8a70bb" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41397 } }
I20260807 03:09:30.350004 25518 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:09:30.350185 25518 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:09:30.350399 25518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:30.352768 25518 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd16109787144a5d8e7bb7550b8a70bb" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41397 } }
I20260807 03:09:30.353261 25518 leader_election.cc:302] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd16109787144a5d8e7bb7550b8a70bb; no voters:
I20260807 03:09:30.353858 25518 leader_election.cc:288] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:09:30.353963 25523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:30.354718 25523 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [term 1 LEADER]: Becoming Leader. State: Replica: dd16109787144a5d8e7bb7550b8a70bb, State: Running, Role: LEADER
I20260807 03:09:30.355309 25523 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd16109787144a5d8e7bb7550b8a70bb" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41397 } }
I20260807 03:09:30.356410 25518 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:09:30.358070 25524 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd16109787144a5d8e7bb7550b8a70bb. Latest consensus state: current_term: 1 leader_uuid: "dd16109787144a5d8e7bb7550b8a70bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd16109787144a5d8e7bb7550b8a70bb" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41397 } } }
I20260807 03:09:30.358640 25524 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:30.358071 25525 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd16109787144a5d8e7bb7550b8a70bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd16109787144a5d8e7bb7550b8a70bb" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41397 } } }
I20260807 03:09:30.358912 25525 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:30.361296 25528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:09:30.366202 25528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:09:30.374627 25528 catalog_manager.cc:1347] Generated new cluster ID: 613754336cb2493dbce78d29004cb576
I20260807 03:09:30.374733 25528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:09:30.384532 25528 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:09:30.385728 25528 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:09:30.399377 25528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb: Generated new TSK 0
I20260807 03:09:30.400179 25528 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.1.keytab.
Entry for principal oryx/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.1.keytab.
I20260807 03:09:30.463716 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:41397
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.1.keytab
--principal=oryx/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:30.808923 25544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:30.809185 25544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:30.809275 25544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:30.816313 25544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:30.816419 25544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:30.816499 25544 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 03:09:30.816579 25544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:09:30.825887 25544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.1.keytab
--principal=oryx/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.31.62:41397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:30.827109 25544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:30.828426 25544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:30.840965 25554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:30.841032 25551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:30.841001 25552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:30.842018 25544 server_base.cc:1034] running on GCE node
I20260807 03:09:30.842656 25544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:30.843493 25544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:30.844732 25544 hybrid_clock.cc:648] HybridClock initialized: now 1786072170844681 us; error 51 us; skew 500 ppm
I20260807 03:09:30.845021 25544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:30 dist-test-slave-tm1g krb5kdc[25420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072170, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:30.848037 25544 init.cc:377] Logged in from keytab as oryx/127.17.31.1@KRBTEST.COM (short username oryx)
I20260807 03:09:30.849407 25544 webserver.cc:460] Webserver started at http://127.17.31.1:37571/ using document root <none> and password file <none>
I20260807 03:09:30.850070 25544 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:30.850199 25544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:30.850575 25544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:30.853389 25544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "516efccf09904a4c9fb3b54576bfd035"
format_stamp: "Formatted at 2026-08-07 03:09:30 on dist-test-slave-tm1g"
server_key: "c780efbd8baf57e3599ea366b53e2ac0"
server_key_iv: "401300765b227b4e9eb1a3845afc114b"
server_key_version: "encryptionkey@0"
I20260807 03:09:30.854157 25544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "516efccf09904a4c9fb3b54576bfd035"
format_stamp: "Formatted at 2026-08-07 03:09:30 on dist-test-slave-tm1g"
server_key: "c780efbd8baf57e3599ea366b53e2ac0"
server_key_iv: "401300765b227b4e9eb1a3845afc114b"
server_key_version: "encryptionkey@0"
I20260807 03:09:30.859187 25544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 03:09:30.862480 25562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:30.863884 25544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 03:09:30.864054 25544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "516efccf09904a4c9fb3b54576bfd035"
format_stamp: "Formatted at 2026-08-07 03:09:30 on dist-test-slave-tm1g"
server_key: "c780efbd8baf57e3599ea366b53e2ac0"
server_key_iv: "401300765b227b4e9eb1a3845afc114b"
server_key_version: "encryptionkey@0"
I20260807 03:09:30.864260 25544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:30.883383 25544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:30.887313 25544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:30.887590 25544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:30.890187 25544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:30.890353 25544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:30.890516 25544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:30.890600 25544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 03:09:30 dist-test-slave-tm1g krb5kdc[25420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072170, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM
I20260807 03:09:30.911921 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:30.890543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:59217 (local address 127.17.31.62:41397)
0807 03:09:30.891076 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:30.891083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:30.892289 (+ 1206us) server_negotiation.cc:407] Connection header received
0807 03:09:30.894319 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:30.894326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:30.894421 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:30.894569 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:30.896803 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:30.898143 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:30.900110 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:30.900929 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:30.901089 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:30.906093 (+ 5004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:30.906098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:30.906102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:30.906133 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:30.908945 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:30.909648 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:30.909659 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:30.909662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:30.909774 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:30.910199 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:30.910204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:30.910209 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:30.910532 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:30.910688 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:30.911339 (+ 651us) server_negotiation.cc:299] Negotiation successful
0807 03:09:30.911551 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":232,"threads_started":1}
I20260807 03:09:30.912909 25573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:30.891197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:41397 (local address 127.17.31.1:59217)
0807 03:09:30.892084 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:30.892135 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:30.894007 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:30.894859 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:30.894870 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:30.895724 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:30.896587 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:30.896621 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:30.898365 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:30.898370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:30.899965 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:30.899976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:30.901312 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:30.901327 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:30.901560 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:30.902328 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:30.902375 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:30.905860 (+ 3485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:30.909089 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:30.909097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:30.909112 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:30.909520 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:30.909884 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:30.909889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:30.909893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:30.910081 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:30.910664 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:30.910671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:30.910995 (+ 324us) client_negotiation.cc:769] Sending connection context
0807 03:09:30.911251 (+ 256us) client_negotiation.cc:240] Negotiation successful
0807 03:09:30.911746 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":368,"threads_started":1}
I20260807 03:09:30.937917 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:30.921107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:41397 (local address 127.17.31.1:40487)
0807 03:09:30.921915 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:30.921939 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:30.922100 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:30.922552 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:30.922557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:30.922984 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:30.923263 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:30.923272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:30.924791 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:30.924796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:30.927015 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:30.927032 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:30.928193 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:30.928202 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:30.928420 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:30.929247 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:30.929278 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:30.930637 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:30.934740 (+ 4103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:30.934748 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:30.934754 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:30.935272 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:30.935806 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:30.935816 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:30.935819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:30.936004 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:30.936701 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:30.936708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:30.936953 (+ 245us) client_negotiation.cc:769] Sending connection context
0807 03:09:30.937244 (+ 291us) client_negotiation.cc:240] Negotiation successful
0807 03:09:30.937502 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":455,"threads_started":1}
I20260807 03:09:30.938191 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:30.921076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:40487 (local address 127.17.31.62:41397)
0807 03:09:30.921268 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:30.921276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:30.922047 (+ 771us) server_negotiation.cc:407] Connection header received
0807 03:09:30.922237 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:30.922245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:30.922362 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:30.922487 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:30.923404 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:30.924658 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:30.927190 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:30.927979 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:30.928138 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:30.930867 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:30.930875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:30.930882 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:30.930925 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:30.934601 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:30.935397 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:30.935405 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:30.935411 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:30.935611 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:30.936126 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:30.936134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:30.936140 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:30.936544 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:30.936725 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:30.937420 (+ 695us) server_negotiation.cc:299] Negotiation successful
0807 03:09:30.937729 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 03:09:30.942592 25544 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:37791
I20260807 03:09:30.942631 25694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:37791 every 8 connection(s)
I20260807 03:09:30.943758 25544 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:09:30.945629 25695 heartbeater.cc:347] Connected to a master server at 127.17.31.62:41397
I20260807 03:09:30.945815 25695 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:30.946453 25695 heartbeater.cc:511] Master 127.17.31.62:41397 requested a full tablet report, sending...
I20260807 03:09:30.948190 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 25544
I20260807 03:09:30.948217 25469 ts_manager.cc:194] Registered new tserver with Master: 516efccf09904a4c9fb3b54576bfd035 (127.17.31.1:37791)
I20260807 03:09:30.948411 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:09:30.949049 17532 external_mini_cluster.cc:1428] Setting key edaac597a1857dc973b4894c9f1400ea
I20260807 03:09:30.949757 25469 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:59217
WARNING: no policy specified for oryx/127.17.31.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.2.keytab.
Entry for principal oryx/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.2.keytab.
I20260807 03:09:30.995390 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:41397
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.2.keytab
--principal=oryx/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:31.311408 25700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:31.311718 25700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:31.311894 25700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:31.319036 25700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:31.319173 25700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:31.319308 25700 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 03:09:31.319443 25700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:09:31.328765 25700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.2.keytab
--principal=oryx/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.31.62:41397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:31.329913 25700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:31.331218 25700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:31.344318 25709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:31.344358 25706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:31.344280 25707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:31.345245 25700 server_base.cc:1034] running on GCE node
I20260807 03:09:31.345830 25700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:31.346810 25700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:31.348044 25700 hybrid_clock.cc:648] HybridClock initialized: now 1786072171347989 us; error 33 us; skew 500 ppm
I20260807 03:09:31.348327 25700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:31 dist-test-slave-tm1g krb5kdc[25420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072171, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:31.351310 25700 init.cc:377] Logged in from keytab as oryx/127.17.31.2@KRBTEST.COM (short username oryx)
I20260807 03:09:31.352680 25700 webserver.cc:460] Webserver started at http://127.17.31.2:40931/ using document root <none> and password file <none>
I20260807 03:09:31.353353 25700 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:31.353492 25700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:31.353804 25700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:31.356616 25700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "c6c7a4bb219f4c109247f187f2cc2e04"
format_stamp: "Formatted at 2026-08-07 03:09:31 on dist-test-slave-tm1g"
server_key: "85bce3c8f4febed13b1d97ad12f30609"
server_key_iv: "9a2d4abf29388f3ae69a96cf5eca0d27"
server_key_version: "encryptionkey@0"
I20260807 03:09:31.357498 25700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "c6c7a4bb219f4c109247f187f2cc2e04"
format_stamp: "Formatted at 2026-08-07 03:09:31 on dist-test-slave-tm1g"
server_key: "85bce3c8f4febed13b1d97ad12f30609"
server_key_iv: "9a2d4abf29388f3ae69a96cf5eca0d27"
server_key_version: "encryptionkey@0"
I20260807 03:09:31.362488 25700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 03:09:31.365710 25717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:31.366964 25700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:09:31.367131 25700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "c6c7a4bb219f4c109247f187f2cc2e04"
format_stamp: "Formatted at 2026-08-07 03:09:31 on dist-test-slave-tm1g"
server_key: "85bce3c8f4febed13b1d97ad12f30609"
server_key_iv: "9a2d4abf29388f3ae69a96cf5eca0d27"
server_key_version: "encryptionkey@0"
I20260807 03:09:31.367344 25700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:31.385092 25700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:31.388772 25700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:31.389040 25700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:31.391611 25700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:31.391732 25700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:31.391896 25700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:31.391963 25700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 03:09:31 dist-test-slave-tm1g krb5kdc[25420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072171, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM
I20260807 03:09:31.413820 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:31.392004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:44795 (local address 127.17.31.62:41397)
0807 03:09:31.392206 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:31.392212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:31.393435 (+ 1223us) server_negotiation.cc:407] Connection header received
0807 03:09:31.395458 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:31.395463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:31.395590 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:31.395701 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:31.397570 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:31.398646 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:31.401081 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:31.401667 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:31.401802 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:31.406169 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:31.406174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:31.406177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:31.406209 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:31.409601 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:31.410796 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:31.410803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:31.410808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:31.410973 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:31.411584 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:31.411593 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:31.411598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:31.412087 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:31.412296 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:31.412988 (+ 692us) server_negotiation.cc:299] Negotiation successful
0807 03:09:31.413299 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 03:09:31.414551 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:31.392369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:41397 (local address 127.17.31.2:44795)
0807 03:09:31.393244 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:31.393294 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:31.395184 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:31.395920 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:31.395928 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:31.396629 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:31.397391 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:31.397413 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:31.398810 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:31.398814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:31.400896 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:31.400916 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:31.401935 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:31.401942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:31.402184 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:31.402893 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:31.402929 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:31.405960 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:31.409757 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:31.409769 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:31.409790 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:31.410638 (+ 848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:31.411100 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:31.411106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:31.411110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:31.411418 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:31.412226 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:31.412231 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:31.412531 (+ 300us) client_negotiation.cc:769] Sending connection context
0807 03:09:31.412852 (+ 321us) client_negotiation.cc:240] Negotiation successful
0807 03:09:31.413299 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":396,"threads_started":1}
I20260807 03:09:31.433506 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:31.420389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:51573 (local address 127.17.31.62:41397)
0807 03:09:31.420541 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:31.420545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:31.421124 (+ 579us) server_negotiation.cc:407] Connection header received
0807 03:09:31.421298 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:31.421311 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:31.421366 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:31.421463 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:31.422340 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:31.423047 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:31.424343 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:31.424737 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:31.424863 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:31.427283 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:31.427289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:31.427295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:31.427329 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:31.430768 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:31.431615 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:31.431623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:31.431628 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:31.431756 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:31.432095 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:31.432101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:31.432104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:31.432413 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:31.432565 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:31.432953 (+ 388us) server_negotiation.cc:299] Negotiation successful
0807 03:09:31.433175 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 03:09:31.433624 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:31.420385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:41397 (local address 127.17.31.2:51573)
0807 03:09:31.421027 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:31.421045 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:31.421189 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:31.421486 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:31.421490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:31.421908 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:31.422189 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:31.422197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:31.423177 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:31.423183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:31.424211 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:31.424222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:31.424943 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:31.424952 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:31.425129 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:31.425868 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:31.425884 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:31.427021 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:31.430904 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:31.430909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:31.430916 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:31.431475 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:31.431863 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:31.431867 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:31.431871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:31.431992 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:31.432538 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:31.432542 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:31.432712 (+ 170us) client_negotiation.cc:769] Sending connection context
0807 03:09:31.432864 (+ 152us) client_negotiation.cc:240] Negotiation successful
0807 03:09:31.433175 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":389,"threads_started":1}
I20260807 03:09:31.446517 25700 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:40519
I20260807 03:09:31.446560 25848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:40519 every 8 connection(s)
I20260807 03:09:31.447669 25700 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:09:31.449482 25849 heartbeater.cc:347] Connected to a master server at 127.17.31.62:41397
I20260807 03:09:31.449712 25849 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:31.450410 25849 heartbeater.cc:511] Master 127.17.31.62:41397 requested a full tablet report, sending...
I20260807 03:09:31.451648 25468 ts_manager.cc:194] Registered new tserver with Master: c6c7a4bb219f4c109247f187f2cc2e04 (127.17.31.2:40519)
I20260807 03:09:31.452620 25468 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:44795
I20260807 03:09:31.455216 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 25700
I20260807 03:09:31.455370 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:09:31.455745 17532 external_mini_cluster.cc:1428] Setting key af96c9e2ded494fb1137bd8738d92c23
WARNING: no policy specified for oryx/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.3.keytab.
Entry for principal oryx/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.3.keytab.
I20260807 03:09:31.503441 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:41397
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.3.keytab
--principal=oryx/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:31.828341 25853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:31.828622 25853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:31.828743 25853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:31.835959 25853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:31.836059 25853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:31.836232 25853 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 03:09:31.836364 25853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:09:31.846256 25853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.3.keytab
--principal=oryx/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.31.62:41397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:31.847780 25853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:31.849156 25853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:31.862774 25858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:31.862776 25859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:31.862702 25861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:31.863894 25853 server_base.cc:1034] running on GCE node
I20260807 03:09:31.864522 25853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:31.865537 25853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:31.866779 25853 hybrid_clock.cc:648] HybridClock initialized: now 1786072171866715 us; error 61 us; skew 500 ppm
I20260807 03:09:31.867089 25853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:31 dist-test-slave-tm1g krb5kdc[25420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072171, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:31.870399 25853 init.cc:377] Logged in from keytab as oryx/127.17.31.3@KRBTEST.COM (short username oryx)
I20260807 03:09:31.871850 25853 webserver.cc:460] Webserver started at http://127.17.31.3:41417/ using document root <none> and password file <none>
I20260807 03:09:31.872539 25853 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:31.872625 25853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:31.872893 25853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:31.875790 25853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "247d9927b6344d729e0ddfacda327123"
format_stamp: "Formatted at 2026-08-07 03:09:31 on dist-test-slave-tm1g"
server_key: "08205e187b2770c5f18c5f9a5dd1bf1f"
server_key_iv: "b7b87accee49473fa6040b0f35da05d4"
server_key_version: "encryptionkey@0"
I20260807 03:09:31.876530 25853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "247d9927b6344d729e0ddfacda327123"
format_stamp: "Formatted at 2026-08-07 03:09:31 on dist-test-slave-tm1g"
server_key: "08205e187b2770c5f18c5f9a5dd1bf1f"
server_key_iv: "b7b87accee49473fa6040b0f35da05d4"
server_key_version: "encryptionkey@0"
I20260807 03:09:31.881345 25853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:09:31.884799 25868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:31.886027 25853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:09:31.886157 25853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "247d9927b6344d729e0ddfacda327123"
format_stamp: "Formatted at 2026-08-07 03:09:31 on dist-test-slave-tm1g"
server_key: "08205e187b2770c5f18c5f9a5dd1bf1f"
server_key_iv: "b7b87accee49473fa6040b0f35da05d4"
server_key_version: "encryptionkey@0"
I20260807 03:09:31.886387 25853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:31.906967 25853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:31.911053 25853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:31.911396 25853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:31.913985 25853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:31.914099 25853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:31.914263 25853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:31.914376 25853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 07 03:09:31 dist-test-slave-tm1g krb5kdc[25420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072171, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM
I20260807 03:09:31.938225 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:31.914818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:54299 (local address 127.17.31.62:41397)
0807 03:09:31.915025 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:31.915031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:31.916598 (+ 1567us) server_negotiation.cc:407] Connection header received
0807 03:09:31.918721 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:31.918728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:31.918822 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:31.918960 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:31.921062 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:31.922421 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:31.924907 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:31.925362 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:31.925499 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:31.930387 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:31.930395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:31.930402 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:31.930451 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:31.934348 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:31.935320 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:31.935330 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:31.935332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:31.935459 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:31.936079 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:31.936088 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:31.936092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:31.936389 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:31.936573 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:31.937635 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0807 03:09:31.937861 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 03:09:31.939247 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:31.915452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:41397 (local address 127.17.31.3:54299)
0807 03:09:31.916428 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:31.916473 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:31.918447 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:31.919253 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:31.919264 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:31.920044 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:31.920850 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:31.920868 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:31.922597 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:31.922606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:31.924720 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:31.924735 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:31.925598 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:31.925605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:31.925824 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:31.926490 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:31.926511 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:31.930101 (+ 3590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:31.934509 (+ 4408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:31.934518 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:31.934539 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:31.935128 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:31.935627 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:31.935632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:31.935638 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:31.935945 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:31.936667 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:31.936676 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:31.937139 (+ 463us) client_negotiation.cc:769] Sending connection context
0807 03:09:31.937549 (+ 410us) client_negotiation.cc:240] Negotiation successful
0807 03:09:31.938072 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":470,"threads_started":1}
I20260807 03:09:31.952459 25695 heartbeater.cc:503] Master 127.17.31.62:41397 was elected leader, sending a full tablet report...
I20260807 03:09:31.960646 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:31.946988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:40453 (local address 127.17.31.62:41397)
0807 03:09:31.947161 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:31.947165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:31.947723 (+ 558us) server_negotiation.cc:407] Connection header received
0807 03:09:31.947940 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:31.947946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:31.948023 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:31.948124 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:31.949362 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:31.950406 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:31.952649 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:31.953199 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:31.953346 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:31.955291 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:31.955296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:31.955301 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:31.955347 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:31.958014 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:31.958759 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:31.958766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:31.958770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:31.958898 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:31.959248 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:31.959254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:31.959257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:31.959523 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:31.959676 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:31.960060 (+ 384us) server_negotiation.cc:299] Negotiation successful
0807 03:09:31.960278 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:09:31.960724 25945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:31.946986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:41397 (local address 127.17.31.3:40453)
0807 03:09:31.947618 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:31.947633 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:31.947800 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:31.948151 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:31.948156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:31.948852 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:31.949216 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:31.949234 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:31.950555 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:31.950560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:31.952521 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:31.952533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:31.953371 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:31.953377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:31.953537 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:31.954236 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:31.954264 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:31.955103 (+ 839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:31.958124 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:31.958130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:31.958134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:31.958637 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:31.959014 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:31.959018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:31.959022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:31.959143 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:31.959639 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:31.959643 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:31.959818 (+ 175us) client_negotiation.cc:769] Sending connection context
0807 03:09:31.959965 (+ 147us) client_negotiation.cc:240] Negotiation successful
0807 03:09:31.960279 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":381,"threads_started":1}
I20260807 03:09:31.977434 25853 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:33165
I20260807 03:09:31.977479 25994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:33165 every 8 connection(s)
I20260807 03:09:31.978722 25853 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:09:31.980868 25995 heartbeater.cc:347] Connected to a master server at 127.17.31.62:41397
I20260807 03:09:31.981091 25995 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:31.981693 25995 heartbeater.cc:511] Master 127.17.31.62:41397 requested a full tablet report, sending...
I20260807 03:09:31.983019 25468 ts_manager.cc:194] Registered new tserver with Master: 247d9927b6344d729e0ddfacda327123 (127.17.31.3:33165)
I20260807 03:09:31.983919 25468 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:54299
I20260807 03:09:31.984737 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 25853
I20260807 03:09:31.984961 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:09:31.985400 17532 external_mini_cluster.cc:1428] Setting key 220a7432510d5aefdba675b077fb9535
I20260807 03:09:31.988292 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 03:09:32 dist-test-slave-tm1g krb5kdc[25420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 03:09:32 dist-test-slave-tm1g krb5kdc[25420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:32 dist-test-slave-tm1g krb5kdc[25420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 03:09:32.040879 25571 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:32.030717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57878 (local address 127.17.31.62:41397)
0807 03:09:32.030961 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:32.030968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:32.031192 (+ 224us) server_negotiation.cc:407] Connection header received
0807 03:09:32.031350 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:32.031354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:32.031422 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:32.031509 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:32.032835 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.033569 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.034877 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.035291 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.035382 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.040640 (+ 5258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57878: BlockingRecv error: recv got EOF from 127.0.0.1:57878 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 07 03:09:32 dist-test-slave-tm1g krb5kdc[25420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM
I20260807 03:09:32.055979 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.043928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.17.31.62:41397)
0807 03:09:32.044143 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:32.044149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:32.044368 (+ 219us) server_negotiation.cc:407] Connection header received
0807 03:09:32.044484 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:32.044488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:32.044558 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:32.044633 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:32.045707 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.046477 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.047613 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.047999 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.048093 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.051505 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:32.051510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:32.051513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:32.051542 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:32.053915 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:32.054534 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:32.054538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:32.054540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:32.054651 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:32.054959 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:32.054962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:32.054964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:32.055219 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:32.055424 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:32.055601 (+ 177us) server_negotiation.cc:299] Negotiation successful
0807 03:09:32.055756 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 03:09:32.059669 25468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:09:32.066591 25468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:09:32.090727 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.080214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:37791 (local address 127.0.0.1:38252)
0807 03:09:32.081181 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:32.081196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:32.081297 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:32.083020 (+ 1723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:32.083027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:32.083064 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:32.083539 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.083551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.085302 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.085308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.087266 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.087280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.089508 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.089517 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.089749 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.089825 (+ 76us) client_negotiation.cc:769] Sending connection context
0807 03:09:32.089975 (+ 150us) client_negotiation.cc:240] Negotiation successful
0807 03:09:32.090191 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":98,"spinlock_wait_cycles":1408,"thread_start_us":723,"threads_started":1}
I20260807 03:09:32.091142 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.078424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:33165 (local address 127.0.0.1:44358)
0807 03:09:32.079840 (+ 1416us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:32.079874 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:32.080029 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:32.082913 (+ 2884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:32.082917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:32.082941 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:32.083489 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.083501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.085731 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.085736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.087996 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.088008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.090046 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.090051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.090245 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.090273 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 03:09:32.090447 (+ 174us) client_negotiation.cc:240] Negotiation successful
0807 03:09:32.090638 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1183,"mutex_wait_us":9,"thread_start_us":1888,"threads_started":1}
I20260807 03:09:32.091558 26016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.078588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:40519 (local address 127.0.0.1:43488)
0807 03:09:32.080588 (+ 2000us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:32.080608 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:32.080715 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:32.082460 (+ 1745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:32.082466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:32.082508 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:32.082917 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.082928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.085000 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.085006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.087597 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.087616 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.089815 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.089822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.090039 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.090765 (+ 726us) client_negotiation.cc:769] Sending connection context
0807 03:09:32.091014 (+ 249us) client_negotiation.cc:240] Negotiation successful
0807 03:09:32.091163 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1858,"mutex_wait_us":715,"thread_start_us":879,"threads_started":1}
I20260807 03:09:32.092129 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.080718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38252 (local address 127.17.31.1:37791)
0807 03:09:32.082476 (+ 1758us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:32.082484 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:32.082507 (+ 23us) server_negotiation.cc:407] Connection header received
0807 03:09:32.082605 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:32.082611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:32.082813 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:32.083510 (+ 697us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:32.083723 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.085142 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.087446 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.088955 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.089108 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.089353 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:32.091589 (+ 2236us) server_negotiation.cc:299] Negotiation successful
0807 03:09:32.091793 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1620,"thread_start_us":1462,"threads_started":1}
I20260807 03:09:32.092778 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.078884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.17.31.3:33165)
0807 03:09:32.082249 (+ 3365us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:32.082258 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:32.082293 (+ 35us) server_negotiation.cc:407] Connection header received
0807 03:09:32.082415 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:32.082421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:32.082669 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:32.082913 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:32.083739 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.085582 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.088170 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.089803 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.091914 (+ 2111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.092144 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:32.092221 (+ 77us) server_negotiation.cc:299] Negotiation successful
0807 03:09:32.092402 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3188,"thread_start_us":3002,"threads_started":1}
I20260807 03:09:32.093533 26018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.078990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.17.31.2:40519)
0807 03:09:32.081742 (+ 2752us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:32.081750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:32.081778 (+ 28us) server_negotiation.cc:407] Connection header received
0807 03:09:32.081896 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:32.081908 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:32.082194 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:32.083159 (+ 965us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:32.083358 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.084855 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.087783 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.089577 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.090128 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.090344 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:32.093032 (+ 2688us) server_negotiation.cc:299] Negotiation successful
0807 03:09:32.093224 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2592,"thread_start_us":2483,"threads_started":1}
I20260807 03:09:32.096402 25618 tablet_service.cc:1467] Processing CreateTablet for tablet a6cbd61320fc4eefb9efa6cab0568a4b (DEFAULT_TABLE table=test-table [id=89d09efbaaeb497b8d7de4f0c32ff289]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:32.096402 25922 tablet_service.cc:1467] Processing CreateTablet for tablet a6cbd61320fc4eefb9efa6cab0568a4b (DEFAULT_TABLE table=test-table [id=89d09efbaaeb497b8d7de4f0c32ff289]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:32.096840 25773 tablet_service.cc:1467] Processing CreateTablet for tablet a6cbd61320fc4eefb9efa6cab0568a4b (DEFAULT_TABLE table=test-table [id=89d09efbaaeb497b8d7de4f0c32ff289]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:32.097853 25618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6cbd61320fc4eefb9efa6cab0568a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:32.097867 25922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6cbd61320fc4eefb9efa6cab0568a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:32.098001 25773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6cbd61320fc4eefb9efa6cab0568a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:32.111158 26023 tablet_bootstrap.cc:492] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04: Bootstrap starting.
I20260807 03:09:32.114566 26023 tablet_bootstrap.cc:654] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:32.115159 26024 tablet_bootstrap.cc:492] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123: Bootstrap starting.
I20260807 03:09:32.116356 26023 log.cc:824] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:32.118454 26025 tablet_bootstrap.cc:492] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035: Bootstrap starting.
I20260807 03:09:32.119449 26024 tablet_bootstrap.cc:654] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:32.119530 26023 tablet_bootstrap.cc:492] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04: No bootstrap required, opened a new log
I20260807 03:09:32.119779 26023 ts_tablet_manager.cc:1397] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260807 03:09:32.120671 26024 log.cc:824] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:32.121065 26025 tablet_bootstrap.cc:654] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:32.121950 26025 log.cc:824] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:32.125360 26025 tablet_bootstrap.cc:492] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035: No bootstrap required, opened a new log
I20260807 03:09:32.125625 26025 ts_tablet_manager.cc:1397] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20260807 03:09:32.126226 26024 tablet_bootstrap.cc:492] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123: No bootstrap required, opened a new log
I20260807 03:09:32.126552 26024 ts_tablet_manager.cc:1397] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260807 03:09:32.131232 26023 raft_consensus.cc:348] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } }
I20260807 03:09:32.131646 26023 raft_consensus.cc:374] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:32.131744 26023 raft_consensus.cc:729] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c7a4bb219f4c109247f187f2cc2e04, State: Initialized, Role: FOLLOWER
I20260807 03:09:32.132411 26023 consensus_queue.cc:260] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } }
I20260807 03:09:32.132805 26025 raft_consensus.cc:348] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } }
I20260807 03:09:32.133134 26025 raft_consensus.cc:374] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:32.133252 26025 raft_consensus.cc:729] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516efccf09904a4c9fb3b54576bfd035, State: Initialized, Role: FOLLOWER
I20260807 03:09:32.133888 25849 heartbeater.cc:503] Master 127.17.31.62:41397 was elected leader, sending a full tablet report...
I20260807 03:09:32.133929 26025 consensus_queue.cc:260] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } }
I20260807 03:09:32.134411 26023 ts_tablet_manager.cc:1428] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260807 03:09:32.135681 26025 ts_tablet_manager.cc:1428] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260807 03:09:32.137050 26024 raft_consensus.cc:348] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } }
I20260807 03:09:32.137405 26024 raft_consensus.cc:374] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:32.137532 26024 raft_consensus.cc:729] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247d9927b6344d729e0ddfacda327123, State: Initialized, Role: FOLLOWER
I20260807 03:09:32.138100 26024 consensus_queue.cc:260] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } }
I20260807 03:09:32.139200 25995 heartbeater.cc:503] Master 127.17.31.62:41397 was elected leader, sending a full tablet report...
I20260807 03:09:32.140879 26024 ts_tablet_manager.cc:1428] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260807 03:09:32.196388 25696 tablet.cc:2369] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:09:32.199479 25850 tablet.cc:2369] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:09:32.230377 25997 tablet.cc:2369] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:09:32.241549 26031 raft_consensus.cc:482] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:09:32.241854 26031 raft_consensus.cc:504] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } }
I20260807 03:09:32.243588 26031 leader_election.cc:288] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6c7a4bb219f4c109247f187f2cc2e04 (127.17.31.2:40519), 516efccf09904a4c9fb3b54576bfd035 (127.17.31.1:37791)
I20260807 03:09:32.249406 26018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.244129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:37539 (local address 127.17.31.2:40519)
0807 03:09:32.244334 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:32.244340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:32.244361 (+ 21us) server_negotiation.cc:407] Connection header received
0807 03:09:32.244574 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:32.244580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:32.244666 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:32.244768 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:32.245422 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.246175 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.247755 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.248522 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.248670 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.248772 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:32.249030 (+ 258us) server_negotiation.cc:299] Negotiation successful
0807 03:09:32.249153 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 03:09:32.249445 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.244025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:40519 (local address 127.17.31.3:37539)
0807 03:09:32.244236 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:32.244258 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:32.244376 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:32.244997 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:32.245003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:32.245044 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:32.245262 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.245271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.246344 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.246348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.247594 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.247605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.248678 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.248684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.248873 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.248897 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 03:09:32.249003 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 03:09:32.249141 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":1}
I20260807 03:09:32.250270 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.244729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:44091 (local address 127.17.31.1:37791)
0807 03:09:32.244939 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:32.244947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:32.245395 (+ 448us) server_negotiation.cc:407] Connection header received
0807 03:09:32.245583 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:32.245587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:32.245658 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:32.245755 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:32.246259 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.246986 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.248588 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.249322 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.249475 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.249576 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:32.249914 (+ 338us) server_negotiation.cc:299] Negotiation successful
0807 03:09:32.250032 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:09:32.250356 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.244456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:37791 (local address 127.17.31.3:44091)
0807 03:09:32.245272 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:32.245286 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:32.245440 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:32.245769 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:32.245773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:32.245794 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:32.246107 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.246116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.247120 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.247125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.248442 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.248453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.249543 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.249549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.249751 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.249784 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 03:09:32.249890 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 03:09:32.250008 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":508,"threads_started":1}
I20260807 03:09:32.250521 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cbd61320fc4eefb9efa6cab0568a4b" candidate_uuid: "247d9927b6344d729e0ddfacda327123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" is_pre_election: true
I20260807 03:09:32.251116 25798 raft_consensus.cc:2454] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 247d9927b6344d729e0ddfacda327123 in term 0.
I20260807 03:09:32.251452 25644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cbd61320fc4eefb9efa6cab0568a4b" candidate_uuid: "247d9927b6344d729e0ddfacda327123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516efccf09904a4c9fb3b54576bfd035" is_pre_election: true
I20260807 03:09:32.251755 25871 leader_election.cc:302] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247d9927b6344d729e0ddfacda327123, c6c7a4bb219f4c109247f187f2cc2e04; no voters:
I20260807 03:09:32.251888 25644 raft_consensus.cc:2454] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 247d9927b6344d729e0ddfacda327123 in term 0.
I20260807 03:09:32.252125 26031 raft_consensus.cc:2781] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:09:32.252274 26031 raft_consensus.cc:482] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:09:32.252432 26031 raft_consensus.cc:3037] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:32.254617 26031 raft_consensus.cc:504] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } }
I20260807 03:09:32.255143 26031 leader_election.cc:288] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [CANDIDATE]: Term 1 election: Requested vote from peers c6c7a4bb219f4c109247f187f2cc2e04 (127.17.31.2:40519), 516efccf09904a4c9fb3b54576bfd035 (127.17.31.1:37791)
I20260807 03:09:32.255546 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cbd61320fc4eefb9efa6cab0568a4b" candidate_uuid: "247d9927b6344d729e0ddfacda327123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6c7a4bb219f4c109247f187f2cc2e04"
I20260807 03:09:32.255622 25644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6cbd61320fc4eefb9efa6cab0568a4b" candidate_uuid: "247d9927b6344d729e0ddfacda327123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516efccf09904a4c9fb3b54576bfd035"
I20260807 03:09:32.255725 25798 raft_consensus.cc:3037] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:32.255836 25644 raft_consensus.cc:3037] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:32.258075 25644 raft_consensus.cc:2454] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 247d9927b6344d729e0ddfacda327123 in term 1.
I20260807 03:09:32.258400 25798 raft_consensus.cc:2454] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 247d9927b6344d729e0ddfacda327123 in term 1.
I20260807 03:09:32.258471 25871 leader_election.cc:302] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247d9927b6344d729e0ddfacda327123, 516efccf09904a4c9fb3b54576bfd035; no voters:
I20260807 03:09:32.258767 26031 raft_consensus.cc:2781] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:32.259210 26031 raft_consensus.cc:686] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [term 1 LEADER]: Becoming Leader. State: Replica: 247d9927b6344d729e0ddfacda327123, State: Running, Role: LEADER
I20260807 03:09:32.259657 26031 consensus_queue.cc:237] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } }
I20260807 03:09:32.264179 25468 catalog_manager.cc:5495] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 reported cstate change: term changed from 0 to 1, leader changed from <none> to 247d9927b6344d729e0ddfacda327123 (127.17.31.3). New cstate: current_term: 1 leader_uuid: "247d9927b6344d729e0ddfacda327123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } health_report { overall_health: UNKNOWN } } }
I20260807 03:09:32.293682 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.288206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.17.31.3:33165)
0807 03:09:32.288442 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:32.288448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:32.288520 (+ 72us) server_negotiation.cc:407] Connection header received
0807 03:09:32.288821 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:32.288825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:32.288910 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:32.289061 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:09:32.289442 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.290177 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.291522 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.291957 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.292092 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.292549 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:09:32.293087 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:09:32.293248 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:32.293331 (+ 83us) server_negotiation.cc:299] Negotiation successful
0807 03:09:32.293419 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 03:09:32.301913 25798 raft_consensus.cc:1261] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Refusing update from remote peer 247d9927b6344d729e0ddfacda327123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:09:32.301913 25644 raft_consensus.cc:1261] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Refusing update from remote peer 247d9927b6344d729e0ddfacda327123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:09:32.302911 26033 consensus_queue.cc:1035] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:32.303261 26031 consensus_queue.cc:1035] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:32.316394 26037 mvcc.cc:205] Tried to move back new op lower bound from 7315751617738108928 to 7315751617578844160. Current Snapshot: MvccSnapshot[applied={T|T < 7315751617738108928}]
I20260807 03:09:32.318498 26038 mvcc.cc:205] Tried to move back new op lower bound from 7315751617738108928 to 7315751617578844160. Current Snapshot: MvccSnapshot[applied={T|T < 7315751617738108928}]
I20260807 03:09:32.315470 26036 mvcc.cc:205] Tried to move back new op lower bound from 7315751617738108928 to 7315751617578844160. Current Snapshot: MvccSnapshot[applied={T|T < 7315751617738108928}]
I20260807 03:09:32.343192 25469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:09:32.343596 25469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:09:32.347970 25469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb: Sending DeleteTablet for 3 replicas of tablet a6cbd61320fc4eefb9efa6cab0568a4b
I20260807 03:09:32.349294 25922 tablet_service.cc:1514] Processing DeleteTablet for tablet a6cbd61320fc4eefb9efa6cab0568a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:32 UTC) from {username='oryx', principal='oryx/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:44358
I20260807 03:09:32.349921 25773 tablet_service.cc:1514] Processing DeleteTablet for tablet a6cbd61320fc4eefb9efa6cab0568a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:32 UTC) from {username='oryx', principal='oryx/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:43488
I20260807 03:09:32.350194 26047 tablet_replica.cc:330] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123: stopping tablet replica
I20260807 03:09:32.350641 26047 raft_consensus.cc:2229] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:09:32.351238 25618 tablet_service.cc:1514] Processing DeleteTablet for tablet a6cbd61320fc4eefb9efa6cab0568a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:32 UTC) from {username='oryx', principal='oryx/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:38252
I20260807 03:09:32.351274 26047 raft_consensus.cc:2258] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:32.352223 26050 tablet_replica.cc:330] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035: stopping tablet replica
I20260807 03:09:32.352608 26050 raft_consensus.cc:2229] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:32.353044 26050 raft_consensus.cc:2258] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:32.353971 25468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:09:32.354627 25468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:09:32.354766 26048 tablet_replica.cc:330] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04: stopping tablet replica
I20260807 03:09:32.355125 26048 raft_consensus.cc:2229] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:32.355209 26050 ts_tablet_manager.cc:1905] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:32.355537 26048 raft_consensus.cc:2258] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:32.358263 26047 ts_tablet_manager.cc:1905] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:32.362942 26048 ts_tablet_manager.cc:1905] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:32.363816 25922 tablet_service.cc:1467] Processing CreateTablet for tablet 57af6d0d202f4c1483dc9ef285b05559 (DEFAULT_TABLE table=test-table [id=003ec5707c5847cb93c93e386f331173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:32.364253 25922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57af6d0d202f4c1483dc9ef285b05559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:32.365907 26047 ts_tablet_manager.cc:1918] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:09:32.366183 26047 log.cc:1196] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/a6cbd61320fc4eefb9efa6cab0568a4b
I20260807 03:09:32.366199 26050 ts_tablet_manager.cc:1918] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:09:32.366359 26050 log.cc:1196] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/a6cbd61320fc4eefb9efa6cab0568a4b
I20260807 03:09:32.366911 26050 ts_tablet_manager.cc:1939] T a6cbd61320fc4eefb9efa6cab0568a4b P 516efccf09904a4c9fb3b54576bfd035: Deleting consensus metadata
I20260807 03:09:32.366917 26047 ts_tablet_manager.cc:1939] T a6cbd61320fc4eefb9efa6cab0568a4b P 247d9927b6344d729e0ddfacda327123: Deleting consensus metadata
I20260807 03:09:32.368841 25618 tablet_service.cc:1467] Processing CreateTablet for tablet 57af6d0d202f4c1483dc9ef285b05559 (DEFAULT_TABLE table=test-table [id=003ec5707c5847cb93c93e386f331173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:32.369239 25618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57af6d0d202f4c1483dc9ef285b05559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:32.369733 26048 ts_tablet_manager.cc:1918] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:09:32.369902 26048 log.cc:1196] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/a6cbd61320fc4eefb9efa6cab0568a4b
I20260807 03:09:32.370455 26048 ts_tablet_manager.cc:1939] T a6cbd61320fc4eefb9efa6cab0568a4b P c6c7a4bb219f4c109247f187f2cc2e04: Deleting consensus metadata
I20260807 03:09:32.372165 25452 catalog_manager.cc:4841] TS c6c7a4bb219f4c109247f187f2cc2e04 (127.17.31.2:40519): tablet a6cbd61320fc4eefb9efa6cab0568a4b (table test-table [id=89d09efbaaeb497b8d7de4f0c32ff289]) successfully deleted
I20260807 03:09:32.372229 25773 tablet_service.cc:1467] Processing CreateTablet for tablet 57af6d0d202f4c1483dc9ef285b05559 (DEFAULT_TABLE table=test-table [id=003ec5707c5847cb93c93e386f331173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:32.372624 25773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57af6d0d202f4c1483dc9ef285b05559. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:32.375465 26025 tablet_bootstrap.cc:492] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035: Bootstrap starting.
I20260807 03:09:32.376125 25453 catalog_manager.cc:4841] TS 247d9927b6344d729e0ddfacda327123 (127.17.31.3:33165): tablet a6cbd61320fc4eefb9efa6cab0568a4b (table test-table [id=89d09efbaaeb497b8d7de4f0c32ff289]) successfully deleted
I20260807 03:09:32.377262 26025 tablet_bootstrap.cc:654] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:32.378679 26023 tablet_bootstrap.cc:492] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04: Bootstrap starting.
I20260807 03:09:32.378731 25452 catalog_manager.cc:4841] TS 516efccf09904a4c9fb3b54576bfd035 (127.17.31.1:37791): tablet a6cbd61320fc4eefb9efa6cab0568a4b (table test-table [id=89d09efbaaeb497b8d7de4f0c32ff289]) successfully deleted
I20260807 03:09:32.379873 26024 tablet_bootstrap.cc:492] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123: Bootstrap starting.
I20260807 03:09:32.380214 26023 tablet_bootstrap.cc:654] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:32.381726 26024 tablet_bootstrap.cc:654] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:32.391216 26025 tablet_bootstrap.cc:492] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035: No bootstrap required, opened a new log
I20260807 03:09:32.391424 26025 ts_tablet_manager.cc:1397] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.011s
I20260807 03:09:32.392284 26025 raft_consensus.cc:348] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.392618 26025 raft_consensus.cc:374] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:32.392757 26025 raft_consensus.cc:729] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516efccf09904a4c9fb3b54576bfd035, State: Initialized, Role: FOLLOWER
I20260807 03:09:32.393137 26025 consensus_queue.cc:260] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.395295 26024 tablet_bootstrap.cc:492] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123: No bootstrap required, opened a new log
I20260807 03:09:32.395479 26024 ts_tablet_manager.cc:1397] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260807 03:09:32.396682 26024 raft_consensus.cc:348] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.396901 26025 ts_tablet_manager.cc:1428] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260807 03:09:32.396996 26024 raft_consensus.cc:374] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:32.397114 26024 raft_consensus.cc:729] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247d9927b6344d729e0ddfacda327123, State: Initialized, Role: FOLLOWER
I20260807 03:09:32.397486 26024 consensus_queue.cc:260] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.401090 26024 ts_tablet_manager.cc:1428] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260807 03:09:32.402704 26023 tablet_bootstrap.cc:492] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04: No bootstrap required, opened a new log
I20260807 03:09:32.402880 26023 ts_tablet_manager.cc:1397] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.016s
I20260807 03:09:32.403515 26023 raft_consensus.cc:348] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.403704 26023 raft_consensus.cc:374] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:32.403800 26023 raft_consensus.cc:729] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c7a4bb219f4c109247f187f2cc2e04, State: Initialized, Role: FOLLOWER
I20260807 03:09:32.404076 26023 consensus_queue.cc:260] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.404565 26023 ts_tablet_manager.cc:1428] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260807 03:09:32.413659 26033 raft_consensus.cc:482] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:09:32.413875 26033 raft_consensus.cc:504] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.414609 26033 leader_election.cc:288] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516efccf09904a4c9fb3b54576bfd035 (127.17.31.1:37791), c6c7a4bb219f4c109247f187f2cc2e04 (127.17.31.2:40519)
I20260807 03:09:32.415091 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57af6d0d202f4c1483dc9ef285b05559" candidate_uuid: "247d9927b6344d729e0ddfacda327123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" is_pre_election: true
I20260807 03:09:32.415084 25644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57af6d0d202f4c1483dc9ef285b05559" candidate_uuid: "247d9927b6344d729e0ddfacda327123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516efccf09904a4c9fb3b54576bfd035" is_pre_election: true
I20260807 03:09:32.415313 25798 raft_consensus.cc:2454] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 247d9927b6344d729e0ddfacda327123 in term 0.
I20260807 03:09:32.415311 25644 raft_consensus.cc:2454] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 247d9927b6344d729e0ddfacda327123 in term 0.
I20260807 03:09:32.415719 25871 leader_election.cc:302] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247d9927b6344d729e0ddfacda327123, c6c7a4bb219f4c109247f187f2cc2e04; no voters:
I20260807 03:09:32.416023 26033 raft_consensus.cc:2781] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:09:32.416143 26033 raft_consensus.cc:482] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:09:32.416263 26033 raft_consensus.cc:3037] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:32.418638 26033 raft_consensus.cc:504] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.419185 26033 leader_election.cc:288] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [CANDIDATE]: Term 1 election: Requested vote from peers 516efccf09904a4c9fb3b54576bfd035 (127.17.31.1:37791), c6c7a4bb219f4c109247f187f2cc2e04 (127.17.31.2:40519)
I20260807 03:09:32.419641 25644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57af6d0d202f4c1483dc9ef285b05559" candidate_uuid: "247d9927b6344d729e0ddfacda327123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516efccf09904a4c9fb3b54576bfd035"
I20260807 03:09:32.419678 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57af6d0d202f4c1483dc9ef285b05559" candidate_uuid: "247d9927b6344d729e0ddfacda327123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6c7a4bb219f4c109247f187f2cc2e04"
I20260807 03:09:32.419847 25644 raft_consensus.cc:3037] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:32.419855 25798 raft_consensus.cc:3037] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:32.421777 25798 raft_consensus.cc:2454] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 247d9927b6344d729e0ddfacda327123 in term 1.
I20260807 03:09:32.422250 25871 leader_election.cc:302] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247d9927b6344d729e0ddfacda327123, c6c7a4bb219f4c109247f187f2cc2e04; no voters:
I20260807 03:09:32.422619 26033 raft_consensus.cc:2781] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:32.422842 25644 raft_consensus.cc:2454] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 247d9927b6344d729e0ddfacda327123 in term 1.
I20260807 03:09:32.423380 26033 raft_consensus.cc:686] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [term 1 LEADER]: Becoming Leader. State: Replica: 247d9927b6344d729e0ddfacda327123, State: Running, Role: LEADER
I20260807 03:09:32.423636 26033 consensus_queue.cc:237] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.426172 25468 catalog_manager.cc:5495] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 reported cstate change: term changed from 0 to 1, leader changed from <none> to 247d9927b6344d729e0ddfacda327123 (127.17.31.3). New cstate: current_term: 1 leader_uuid: "247d9927b6344d729e0ddfacda327123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } health_report { overall_health: HEALTHY } } }
Aug 07 03:09:32 dist-test-slave-tm1g krb5kdc[25420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072170, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.62@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM
I20260807 03:09:32.483037 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.469232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:41397 (local address 127.0.0.1:57906)
0807 03:09:32.469918 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:32.469942 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:32.470131 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:32.470531 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:32.470535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:32.471363 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:32.471643 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.471652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.472684 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.472692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.473641 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.473652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.474405 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.474411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.474585 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.475255 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:32.475273 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:32.477471 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:32.480344 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:32.480355 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:32.480362 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:32.480767 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:32.481113 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:32.481119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:32.481122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:32.481274 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:32.481937 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:32.481943 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:32.482104 (+ 161us) client_negotiation.cc:769] Sending connection context
0807 03:09:32.482240 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 03:09:32.482524 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":397,"threads_started":1}
I20260807 03:09:32.483168 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.469184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.17.31.62:41397)
0807 03:09:32.469383 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:32.469391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:32.470047 (+ 656us) server_negotiation.cc:407] Connection header received
0807 03:09:32.470285 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:32.470290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:32.470402 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:32.470501 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:32.471780 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.472568 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.473790 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.474210 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.474364 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.477660 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:32.477665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:32.477668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:32.477696 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:32.480201 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:32.480883 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:32.480890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:32.480894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:32.481007 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:32.481433 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:32.481442 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:32.481446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:32.481784 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:32.481978 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:32.482395 (+ 417us) server_negotiation.cc:299] Negotiation successful
0807 03:09:32.482695 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":66}
I20260807 03:09:32.488198 25468 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:57906:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 03:09:32.489459 25468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:09:32.496407 25773 tablet_service.cc:1467] Processing CreateTablet for tablet d048872f883b428fb4b14bfcb7d7911f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7ce78cd2f14748f8aab8db8b009bda7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 03:09:32.496407 25618 tablet_service.cc:1467] Processing CreateTablet for tablet d048872f883b428fb4b14bfcb7d7911f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7ce78cd2f14748f8aab8db8b009bda7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 03:09:32.496968 25773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d048872f883b428fb4b14bfcb7d7911f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:32.496968 25618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d048872f883b428fb4b14bfcb7d7911f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:32.497078 25922 tablet_service.cc:1467] Processing CreateTablet for tablet d048872f883b428fb4b14bfcb7d7911f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7ce78cd2f14748f8aab8db8b009bda7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 03:09:32.497615 25922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d048872f883b428fb4b14bfcb7d7911f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:32.504235 26023 tablet_bootstrap.cc:492] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04: Bootstrap starting.
I20260807 03:09:32.504385 26025 tablet_bootstrap.cc:492] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035: Bootstrap starting.
I20260807 03:09:32.505980 26025 tablet_bootstrap.cc:654] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:32.505980 26023 tablet_bootstrap.cc:654] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:32.507117 26024 tablet_bootstrap.cc:492] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123: Bootstrap starting.
I20260807 03:09:32.509057 26024 tablet_bootstrap.cc:654] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:32.509662 26023 tablet_bootstrap.cc:492] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04: No bootstrap required, opened a new log
I20260807 03:09:32.509833 26023 ts_tablet_manager.cc:1397] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 03:09:32.510818 26023 raft_consensus.cc:348] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.511145 26023 raft_consensus.cc:374] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:32.511230 26023 raft_consensus.cc:729] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c7a4bb219f4c109247f187f2cc2e04, State: Initialized, Role: FOLLOWER
I20260807 03:09:32.511701 26025 tablet_bootstrap.cc:492] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035: No bootstrap required, opened a new log
I20260807 03:09:32.511581 26023 consensus_queue.cc:260] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.511974 26025 ts_tablet_manager.cc:1397] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20260807 03:09:32.512131 26029 tablet_replica.cc:439] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } }
I20260807 03:09:32.512516 26029 tablet_replica.cc:442] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:09:32.512876 26025 raft_consensus.cc:348] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.513171 26025 raft_consensus.cc:374] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:32.513290 26025 raft_consensus.cc:729] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516efccf09904a4c9fb3b54576bfd035, State: Initialized, Role: FOLLOWER
I20260807 03:09:32.513554 26025 consensus_queue.cc:260] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.514037 26030 tablet_replica.cc:439] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } }
I20260807 03:09:32.514359 26030 tablet_replica.cc:442] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:09:32.515902 26025 ts_tablet_manager.cc:1428] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260807 03:09:32.516916 26023 ts_tablet_manager.cc:1428] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260807 03:09:32.520539 26024 tablet_bootstrap.cc:492] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123: No bootstrap required, opened a new log
I20260807 03:09:32.520766 26024 ts_tablet_manager.cc:1397] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20260807 03:09:32.521592 26024 raft_consensus.cc:348] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.521811 26024 raft_consensus.cc:374] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:32.521904 26024 raft_consensus.cc:729] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247d9927b6344d729e0ddfacda327123, State: Initialized, Role: FOLLOWER
I20260807 03:09:32.522161 26024 consensus_queue.cc:260] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.522832 26024 ts_tablet_manager.cc:1428] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 03:09:32.522739 26033 tablet_replica.cc:439] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } }
I20260807 03:09:32.523027 26033 tablet_replica.cc:442] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:09:32.910517 26033 consensus_queue.cc:1035] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:32.915390 26033 consensus_queue.cc:1035] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:32.929658 26030 raft_consensus.cc:482] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:09:32.929823 26030 raft_consensus.cc:504] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.931332 26030 leader_election.cc:288] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6c7a4bb219f4c109247f187f2cc2e04 (127.17.31.2:40519), 247d9927b6344d729e0ddfacda327123 (127.17.31.3:33165)
I20260807 03:09:32.937644 26029 raft_consensus.cc:482] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:09:32.937879 26029 raft_consensus.cc:504] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.939880 26029 leader_election.cc:288] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516efccf09904a4c9fb3b54576bfd035 (127.17.31.1:37791), 247d9927b6344d729e0ddfacda327123 (127.17.31.3:33165)
I20260807 03:09:32.958617 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.931790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:41909 (local address 127.17.31.3:33165)
0807 03:09:32.947319 (+ 15529us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:32.947327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:32.949960 (+ 2633us) server_negotiation.cc:407] Connection header received
0807 03:09:32.950059 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:32.950067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:32.950166 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:32.950278 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:32.951646 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.953009 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.955529 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.956748 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.956867 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.957029 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:32.958130 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0807 03:09:32.958280 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15380,"thread_start_us":15272,"threads_started":1}
I20260807 03:09:32.958997 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.931796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:39207 (local address 127.17.31.2:40519)
0807 03:09:32.947715 (+ 15919us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:32.947723 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:32.951383 (+ 3660us) server_negotiation.cc:407] Connection header received
0807 03:09:32.951642 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:32.951647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:32.951741 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:32.951813 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:32.952640 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.953813 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.956353 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.957730 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.957822 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.957900 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:32.958530 (+ 630us) server_negotiation.cc:299] Negotiation successful
0807 03:09:32.958688 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15770,"threads_started":1}
I20260807 03:09:32.958997 26071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.931603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:33165 (local address 127.17.31.1:41909)
0807 03:09:32.949601 (+ 17998us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:32.949634 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:32.949812 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:32.950982 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:32.950987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:32.951045 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:32.951466 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.951476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.953202 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.953207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.955353 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.955368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.957124 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.957133 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.957326 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.957377 (+ 51us) client_negotiation.cc:769] Sending connection context
0807 03:09:32.957474 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 03:09:32.957618 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17814,"mutex_wait_us":1,"thread_start_us":19180,"threads_started":1}
I20260807 03:09:32.959682 26070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.931603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:40519 (local address 127.17.31.1:39207)
0807 03:09:32.951266 (+ 19663us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:32.951284 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:32.951476 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:32.952058 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:32.952062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:32.952087 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:32.952460 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.952471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.953994 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.953999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.956139 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.956154 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.958105 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.958115 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.958337 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.958368 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 03:09:32.959139 (+ 771us) client_negotiation.cc:240] Negotiation successful
0807 03:09:32.959275 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19530,"mutex_wait_us":55,"thread_start_us":18958,"threads_started":1}
I20260807 03:09:32.959775 25948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d048872f883b428fb4b14bfcb7d7911f" candidate_uuid: "516efccf09904a4c9fb3b54576bfd035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247d9927b6344d729e0ddfacda327123" is_pre_election: true
I20260807 03:09:32.960187 25948 raft_consensus.cc:2454] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516efccf09904a4c9fb3b54576bfd035 in term 0.
I20260807 03:09:32.960479 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d048872f883b428fb4b14bfcb7d7911f" candidate_uuid: "516efccf09904a4c9fb3b54576bfd035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" is_pre_election: true
I20260807 03:09:32.960734 25798 raft_consensus.cc:2454] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516efccf09904a4c9fb3b54576bfd035 in term 0.
I20260807 03:09:32.960731 25567 leader_election.cc:302] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247d9927b6344d729e0ddfacda327123, 516efccf09904a4c9fb3b54576bfd035; no voters:
I20260807 03:09:32.961138 26030 raft_consensus.cc:2781] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:09:32.961280 26030 raft_consensus.cc:482] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:09:32.961377 26030 raft_consensus.cc:3037] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:32.964689 26030 raft_consensus.cc:504] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.965608 26030 leader_election.cc:288] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [CANDIDATE]: Term 1 election: Requested vote from peers c6c7a4bb219f4c109247f187f2cc2e04 (127.17.31.2:40519), 247d9927b6344d729e0ddfacda327123 (127.17.31.3:33165)
I20260807 03:09:32.966025 25948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d048872f883b428fb4b14bfcb7d7911f" candidate_uuid: "516efccf09904a4c9fb3b54576bfd035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247d9927b6344d729e0ddfacda327123"
I20260807 03:09:32.966205 25948 raft_consensus.cc:3037] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:32.969230 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.943038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:58773 (local address 127.17.31.3:33165)
0807 03:09:32.958860 (+ 15822us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:32.958867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:32.961904 (+ 3037us) server_negotiation.cc:407] Connection header received
0807 03:09:32.962106 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:32.962111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:32.962179 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:32.962236 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:32.963129 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.964214 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.966750 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.968045 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.968146 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.968239 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:32.968795 (+ 556us) server_negotiation.cc:299] Negotiation successful
0807 03:09:32.968936 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15709,"thread_start_us":19709,"threads_started":1}
I20260807 03:09:32.969259 25948 raft_consensus.cc:2454] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516efccf09904a4c9fb3b54576bfd035 in term 1.
I20260807 03:09:32.969672 25567 leader_election.cc:302] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247d9927b6344d729e0ddfacda327123, 516efccf09904a4c9fb3b54576bfd035; no voters:
I20260807 03:09:32.969934 26074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.940801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:33165 (local address 127.17.31.2:58773)
0807 03:09:32.961825 (+ 21024us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:32.961842 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:32.961976 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:32.962457 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:32.962463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:32.962488 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:32.962938 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.962949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.964384 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.964389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.966575 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.966590 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.968403 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.968409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.968619 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.968651 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 03:09:32.969398 (+ 747us) client_negotiation.cc:240] Negotiation successful
0807 03:09:32.969555 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20917,"mutex_wait_us":2,"thread_start_us":22168,"threads_started":1}
I20260807 03:09:32.969982 26030 raft_consensus.cc:2781] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:32.970047 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.943427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:42805 (local address 127.17.31.1:37791)
0807 03:09:32.944105 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:32.944115 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:32.961802 (+ 17687us) server_negotiation.cc:407] Connection header received
0807 03:09:32.962071 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:32.962077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:32.962144 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:32.962214 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:09:32.963586 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.964865 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.967349 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.968641 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.968751 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.968877 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:32.969513 (+ 636us) server_negotiation.cc:299] Negotiation successful
0807 03:09:32.969667 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":1397,"threads_started":1}
I20260807 03:09:32.970583 25948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d048872f883b428fb4b14bfcb7d7911f" candidate_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247d9927b6344d729e0ddfacda327123" is_pre_election: true
I20260807 03:09:32.970752 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:32.940148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:37791 (local address 127.17.31.2:42805)
0807 03:09:32.961712 (+ 21564us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:32.961730 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:32.961905 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:32.962588 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:32.962597 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:32.962693 (+ 96us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:09:32.963401 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.963414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.965082 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.965087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.967173 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:32.967187 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:32.969065 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:32.969072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:32.969261 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:32.969297 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 03:09:32.970200 (+ 903us) client_negotiation.cc:240] Negotiation successful
0807 03:09:32.970354 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21415,"mutex_wait_us":361,"thread_start_us":22663,"threads_started":1}
I20260807 03:09:32.970872 25948 raft_consensus.cc:2379] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6c7a4bb219f4c109247f187f2cc2e04 in current term 1: Already voted for candidate 516efccf09904a4c9fb3b54576bfd035 in this term.
I20260807 03:09:32.971407 25644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d048872f883b428fb4b14bfcb7d7911f" candidate_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516efccf09904a4c9fb3b54576bfd035" is_pre_election: true
I20260807 03:09:32.971715 25798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d048872f883b428fb4b14bfcb7d7911f" candidate_uuid: "516efccf09904a4c9fb3b54576bfd035" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6c7a4bb219f4c109247f187f2cc2e04"
I20260807 03:09:32.971968 25798 raft_consensus.cc:3037] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:32.972429 26030 raft_consensus.cc:686] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [term 1 LEADER]: Becoming Leader. State: Replica: 516efccf09904a4c9fb3b54576bfd035, State: Running, Role: LEADER
I20260807 03:09:32.972833 26030 consensus_queue.cc:237] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:32.974826 25798 raft_consensus.cc:2454] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516efccf09904a4c9fb3b54576bfd035 in term 1.
I20260807 03:09:32.975622 25721 leader_election.cc:302] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6c7a4bb219f4c109247f187f2cc2e04; no voters: 247d9927b6344d729e0ddfacda327123, 516efccf09904a4c9fb3b54576bfd035
I20260807 03:09:32.975742 26078 tablet_replica.cc:439] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035: TxnStatusTablet state changed. Reason: New leader 516efccf09904a4c9fb3b54576bfd035. Latest consensus state: current_term: 1 leader_uuid: "516efccf09904a4c9fb3b54576bfd035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } }
I20260807 03:09:32.976117 26078 tablet_replica.cc:442] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035: This TxnStatusTablet replica's current role is: LEADER
I20260807 03:09:32.976366 26029 raft_consensus.cc:2726] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:09:32.977475 25469 catalog_manager.cc:5495] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 reported cstate change: term changed from 0 to 1, leader changed from <none> to 516efccf09904a4c9fb3b54576bfd035 (127.17.31.1). New cstate: current_term: 1 leader_uuid: "516efccf09904a4c9fb3b54576bfd035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } health_report { overall_health: UNKNOWN } } }
I20260807 03:09:32.986874 26081 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 03:09:32.987118 26081 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 03:09:33.077064 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:33.071195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.17.31.1:37791)
0807 03:09:33.071412 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:33.071421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:33.071776 (+ 355us) server_negotiation.cc:407] Connection header received
0807 03:09:33.072084 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:33.072089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:33.072178 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:33.072322 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:09:33.072745 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.073516 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.074936 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.075365 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.075509 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:33.075917 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:09:33.076415 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:09:33.076576 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:33.076663 (+ 87us) server_negotiation.cc:299] Negotiation successful
0807 03:09:33.076739 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 03:09:33.077106 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:33.071154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:37791 (local address 127.0.0.1:38268)
0807 03:09:33.071657 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:33.071689 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:33.071860 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:33.072339 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:33.072344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:33.072355 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 03:09:33.072606 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:33.072615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.073677 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.073682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:33.074775 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:33.074787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.075514 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.075520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:33.075678 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:33.075729 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 03:09:33.076529 (+ 800us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 03:09:33.076559 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 03:09:33.076641 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 03:09:33.076769 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":277,"threads_started":1}
I20260807 03:09:33.081251 25798 raft_consensus.cc:1261] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Refusing update from remote peer 516efccf09904a4c9fb3b54576bfd035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:09:33.081472 25948 raft_consensus.cc:1261] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123 [term 1 FOLLOWER]: Refusing update from remote peer 516efccf09904a4c9fb3b54576bfd035: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:09:33.082083 26078 consensus_queue.cc:1035] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:33.082580 26030 consensus_queue.cc:1035] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035 [LEADER]: Connected to new peer: Peer: permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:33.090617 26033 tablet_replica.cc:439] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123: TxnStatusTablet state changed. Reason: New leader 516efccf09904a4c9fb3b54576bfd035. Latest consensus state: current_term: 1 leader_uuid: "516efccf09904a4c9fb3b54576bfd035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } }
I20260807 03:09:33.090910 26033 tablet_replica.cc:442] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:09:33.093005 26030 tablet_replica.cc:439] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "516efccf09904a4c9fb3b54576bfd035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } }
I20260807 03:09:33.093278 26030 tablet_replica.cc:442] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035: This TxnStatusTablet replica's current role is: LEADER
I20260807 03:09:33.097760 26029 tablet_replica.cc:439] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04: TxnStatusTablet state changed. Reason: New leader 516efccf09904a4c9fb3b54576bfd035. Latest consensus state: current_term: 1 leader_uuid: "516efccf09904a4c9fb3b54576bfd035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } }
I20260807 03:09:33.098007 26029 tablet_replica.cc:442] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:09:33.099066 26030 tablet_replica.cc:439] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "516efccf09904a4c9fb3b54576bfd035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } }
I20260807 03:09:33.099280 26030 tablet_replica.cc:442] T d048872f883b428fb4b14bfcb7d7911f P 516efccf09904a4c9fb3b54576bfd035: This TxnStatusTablet replica's current role is: LEADER
I20260807 03:09:33.111626 26029 tablet_replica.cc:439] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "516efccf09904a4c9fb3b54576bfd035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } }
I20260807 03:09:33.111876 26029 tablet_replica.cc:442] T d048872f883b428fb4b14bfcb7d7911f P c6c7a4bb219f4c109247f187f2cc2e04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:09:33.113528 26033 tablet_replica.cc:439] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "516efccf09904a4c9fb3b54576bfd035" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } } }
I20260807 03:09:33.113781 26033 tablet_replica.cc:442] T d048872f883b428fb4b14bfcb7d7911f P 247d9927b6344d729e0ddfacda327123: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 03:09:33.125871 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:33.120482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:48157 (local address 127.17.31.1:37791)
0807 03:09:33.120687 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:33.120696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:33.121186 (+ 490us) server_negotiation.cc:407] Connection header received
0807 03:09:33.121343 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:33.121346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:33.121416 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:33.121486 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:09:33.121988 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.122806 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.124094 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.124545 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.124704 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:33.125129 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:09:33.125230 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:09:33.125342 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:33.125473 (+ 131us) server_negotiation.cc:299] Negotiation successful
0807 03:09:33.125547 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 03:09:33.125924 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:33.120462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:37791 (local address 127.17.31.3:48157)
0807 03:09:33.121077 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:33.121098 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:33.121243 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:33.121538 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:33.121543 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:33.121558 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 03:09:33.121830 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:33.121841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.122938 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.122943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:33.123953 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:33.123964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.124738 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.124746 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:33.124902 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:33.124950 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 03:09:33.125342 (+ 392us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 03:09:33.125356 (+ 14us) client_negotiation.cc:769] Sending connection context
0807 03:09:33.125454 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 03:09:33.125583 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":326,"threads_started":1}
I20260807 03:09:33.164381 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:33.157243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:37309 (local address 127.17.31.3:33165)
0807 03:09:33.157425 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:33.157431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:33.158095 (+ 664us) server_negotiation.cc:407] Connection header received
0807 03:09:33.158282 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:33.158290 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:33.158389 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:33.158484 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:09:33.159162 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.160239 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.162464 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.163028 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.163164 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:33.163794 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:09:33.163906 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:09:33.163973 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:33.164110 (+ 137us) server_negotiation.cc:299] Negotiation successful
0807 03:09:33.164166 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 03:09:33.164894 26098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:33.157208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:33165 (local address 127.17.31.1:37309)
0807 03:09:33.157970 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:33.157986 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:33.158152 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:33.158539 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:33.158545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:33.158558 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 03:09:33.158943 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:33.158967 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.160359 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.160364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:33.162291 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:33.162336 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.163326 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.163332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:33.163502 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:33.163584 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 03:09:33.164001 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 03:09:33.164012 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 03:09:33.164100 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 03:09:33.164244 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":416,"threads_started":1}
I20260807 03:09:33.199258 25469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:09:33.199514 25469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:09:33.202206 25469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb: Sending DeleteTablet for 3 replicas of tablet 57af6d0d202f4c1483dc9ef285b05559
I20260807 03:09:33.202916 25618 tablet_service.cc:1514] Processing DeleteTablet for tablet 57af6d0d202f4c1483dc9ef285b05559 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:33 UTC) from {username='oryx', principal='oryx/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:38252
I20260807 03:09:33.202986 25773 tablet_service.cc:1514] Processing DeleteTablet for tablet 57af6d0d202f4c1483dc9ef285b05559 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:33 UTC) from {username='oryx', principal='oryx/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:43488
I20260807 03:09:33.203123 25922 tablet_service.cc:1514] Processing DeleteTablet for tablet 57af6d0d202f4c1483dc9ef285b05559 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:33 UTC) from {username='oryx', principal='oryx/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:44358
I20260807 03:09:33.216020 26101 tablet_replica.cc:330] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04: stopping tablet replica
I20260807 03:09:33.216400 26101 raft_consensus.cc:2229] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:33.215548 26102 tablet_replica.cc:330] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123: stopping tablet replica
I20260807 03:09:33.216912 26102 raft_consensus.cc:2229] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:09:33.217568 26102 raft_consensus.cc:2258] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:33.217753 26101 raft_consensus.cc:2258] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:33.218044 26100 tablet_replica.cc:330] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035: stopping tablet replica
I20260807 03:09:33.218257 26100 raft_consensus.cc:2229] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:33.218577 26100 raft_consensus.cc:2258] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:33.219719 26100 ts_tablet_manager.cc:1905] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:33.219717 26102 ts_tablet_manager.cc:1905] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:33.220315 26101 ts_tablet_manager.cc:1905] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:33.227437 26100 ts_tablet_manager.cc:1918] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 03:09:33.227581 26100 log.cc:1196] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/57af6d0d202f4c1483dc9ef285b05559
I20260807 03:09:33.228055 26100 ts_tablet_manager.cc:1939] T 57af6d0d202f4c1483dc9ef285b05559 P 516efccf09904a4c9fb3b54576bfd035: Deleting consensus metadata
I20260807 03:09:33.229206 25452 catalog_manager.cc:4841] TS 516efccf09904a4c9fb3b54576bfd035 (127.17.31.1:37791): tablet 57af6d0d202f4c1483dc9ef285b05559 (table test-table [id=003ec5707c5847cb93c93e386f331173]) successfully deleted
I20260807 03:09:33.230851 26101 ts_tablet_manager.cc:1918] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 03:09:33.230978 26101 log.cc:1196] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/57af6d0d202f4c1483dc9ef285b05559
I20260807 03:09:33.231422 26101 ts_tablet_manager.cc:1939] T 57af6d0d202f4c1483dc9ef285b05559 P c6c7a4bb219f4c109247f187f2cc2e04: Deleting consensus metadata
I20260807 03:09:33.232760 25452 catalog_manager.cc:4841] TS c6c7a4bb219f4c109247f187f2cc2e04 (127.17.31.2:40519): tablet 57af6d0d202f4c1483dc9ef285b05559 (table test-table [id=003ec5707c5847cb93c93e386f331173]) successfully deleted
I20260807 03:09:33.235734 26102 ts_tablet_manager.cc:1918] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 03:09:33.235939 26102 log.cc:1196] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/57af6d0d202f4c1483dc9ef285b05559
I20260807 03:09:33.236337 26102 ts_tablet_manager.cc:1939] T 57af6d0d202f4c1483dc9ef285b05559 P 247d9927b6344d729e0ddfacda327123: Deleting consensus metadata
I20260807 03:09:33.237311 25453 catalog_manager.cc:4841] TS 247d9927b6344d729e0ddfacda327123 (127.17.31.3:33165): tablet 57af6d0d202f4c1483dc9ef285b05559 (table test-table [id=003ec5707c5847cb93c93e386f331173]) successfully deleted
Aug 07 03:09:33 dist-test-slave-tm1g krb5kdc[25420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 03:09:33 dist-test-slave-tm1g krb5kdc[25420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:33 dist-test-slave-tm1g krb5kdc[25420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 03:09:33.279342 26111 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:33.262234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.17.31.62:41397)
0807 03:09:33.263048 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:33.263053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:33.263069 (+ 16us) server_negotiation.cc:407] Connection header received
0807 03:09:33.263122 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:33.263125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:33.263182 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:33.263270 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:33.265076 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.265880 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.267860 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.268723 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.268844 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:33.279009 (+ 10165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57920: BlockingRecv error: recv got EOF from 127.0.0.1:57920 (error 108)
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":538,"threads_started":1}
Aug 07 03:09:33 dist-test-slave-tm1g krb5kdc[25420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM
I20260807 03:09:33.297067 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:33.281950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.17.31.62:41397)
0807 03:09:33.282130 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:33.282135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:33.282414 (+ 279us) server_negotiation.cc:407] Connection header received
0807 03:09:33.282594 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:33.282598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:33.282655 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:33.282780 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:33.284398 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.285091 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.286935 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.287334 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.287432 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:33.291873 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:33.291879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:33.291883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:33.291912 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:33.294560 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:33.295170 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:33.295176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:33.295178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:33.295303 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:33.295698 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:33.295702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:33.295704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:33.295956 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:33.296213 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:33.296537 (+ 324us) server_negotiation.cc:299] Negotiation successful
0807 03:09:33.296753 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 03:09:33.300547 25469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:09:33.301149 25469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:09:33.308139 25618 tablet_service.cc:1467] Processing CreateTablet for tablet 62f6de515bd243a1a0fabcecb2af272a (DEFAULT_TABLE table=test-table [id=77dba33d1c104c56852029a5ed3d406c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:33.308553 25922 tablet_service.cc:1467] Processing CreateTablet for tablet 62f6de515bd243a1a0fabcecb2af272a (DEFAULT_TABLE table=test-table [id=77dba33d1c104c56852029a5ed3d406c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:33.308640 25618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f6de515bd243a1a0fabcecb2af272a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:33.308950 25922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f6de515bd243a1a0fabcecb2af272a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:33.313772 26117 tablet_bootstrap.cc:492] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123: Bootstrap starting.
I20260807 03:09:33.315845 26117 tablet_bootstrap.cc:654] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:33.316527 25773 tablet_service.cc:1467] Processing CreateTablet for tablet 62f6de515bd243a1a0fabcecb2af272a (DEFAULT_TABLE table=test-table [id=77dba33d1c104c56852029a5ed3d406c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:33.316958 25773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f6de515bd243a1a0fabcecb2af272a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:33.324301 26117 tablet_bootstrap.cc:492] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123: No bootstrap required, opened a new log
I20260807 03:09:33.324501 26117 ts_tablet_manager.cc:1397] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260807 03:09:33.325282 26117 raft_consensus.cc:348] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.325631 26117 raft_consensus.cc:374] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:33.325759 26117 raft_consensus.cc:729] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247d9927b6344d729e0ddfacda327123, State: Initialized, Role: FOLLOWER
I20260807 03:09:33.326205 26117 consensus_queue.cc:260] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.329149 26117 ts_tablet_manager.cc:1428] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260807 03:09:33.331672 26120 tablet_bootstrap.cc:492] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04: Bootstrap starting.
I20260807 03:09:33.332348 26118 tablet_bootstrap.cc:492] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035: Bootstrap starting.
I20260807 03:09:33.335170 26118 tablet_bootstrap.cc:654] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:33.337970 26118 tablet_bootstrap.cc:492] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035: No bootstrap required, opened a new log
I20260807 03:09:33.338127 26118 ts_tablet_manager.cc:1397] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260807 03:09:33.338340 26120 tablet_bootstrap.cc:654] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:33.339071 26118 raft_consensus.cc:348] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.339351 26118 raft_consensus.cc:374] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:33.339465 26118 raft_consensus.cc:729] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516efccf09904a4c9fb3b54576bfd035, State: Initialized, Role: FOLLOWER
I20260807 03:09:33.339772 26118 consensus_queue.cc:260] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.340432 26118 ts_tablet_manager.cc:1428] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260807 03:09:33.345593 26120 tablet_bootstrap.cc:492] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04: No bootstrap required, opened a new log
I20260807 03:09:33.345748 26120 ts_tablet_manager.cc:1397] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20260807 03:09:33.346558 26120 raft_consensus.cc:348] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.346796 26120 raft_consensus.cc:374] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:33.346890 26120 raft_consensus.cc:729] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c7a4bb219f4c109247f187f2cc2e04, State: Initialized, Role: FOLLOWER
I20260807 03:09:33.347148 26120 consensus_queue.cc:260] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.347843 26120 ts_tablet_manager.cc:1428] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260807 03:09:33.479835 26029 raft_consensus.cc:482] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:09:33.479995 26029 raft_consensus.cc:504] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.480654 26029 leader_election.cc:288] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516efccf09904a4c9fb3b54576bfd035 (127.17.31.1:37791), 247d9927b6344d729e0ddfacda327123 (127.17.31.3:33165)
I20260807 03:09:33.481119 25644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f6de515bd243a1a0fabcecb2af272a" candidate_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516efccf09904a4c9fb3b54576bfd035" is_pre_election: true
I20260807 03:09:33.481158 25948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f6de515bd243a1a0fabcecb2af272a" candidate_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247d9927b6344d729e0ddfacda327123" is_pre_election: true
I20260807 03:09:33.481375 25948 raft_consensus.cc:2454] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c7a4bb219f4c109247f187f2cc2e04 in term 0.
I20260807 03:09:33.481386 25644 raft_consensus.cc:2454] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c7a4bb219f4c109247f187f2cc2e04 in term 0.
I20260807 03:09:33.481812 25721 leader_election.cc:302] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 247d9927b6344d729e0ddfacda327123, 516efccf09904a4c9fb3b54576bfd035, c6c7a4bb219f4c109247f187f2cc2e04; no voters:
I20260807 03:09:33.482086 26029 raft_consensus.cc:2781] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:09:33.482219 26029 raft_consensus.cc:482] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:09:33.482362 26029 raft_consensus.cc:3037] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:33.484566 26029 raft_consensus.cc:504] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.485105 26029 leader_election.cc:288] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [CANDIDATE]: Term 1 election: Requested vote from peers 516efccf09904a4c9fb3b54576bfd035 (127.17.31.1:37791), 247d9927b6344d729e0ddfacda327123 (127.17.31.3:33165)
I20260807 03:09:33.485627 25948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f6de515bd243a1a0fabcecb2af272a" candidate_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247d9927b6344d729e0ddfacda327123"
I20260807 03:09:33.485626 25644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f6de515bd243a1a0fabcecb2af272a" candidate_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516efccf09904a4c9fb3b54576bfd035"
I20260807 03:09:33.485826 25644 raft_consensus.cc:3037] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:33.485826 25948 raft_consensus.cc:3037] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:33.488706 25644 raft_consensus.cc:2454] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c7a4bb219f4c109247f187f2cc2e04 in term 1.
I20260807 03:09:33.488706 25948 raft_consensus.cc:2454] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c7a4bb219f4c109247f187f2cc2e04 in term 1.
I20260807 03:09:33.489121 25721 leader_election.cc:302] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516efccf09904a4c9fb3b54576bfd035, c6c7a4bb219f4c109247f187f2cc2e04; no voters:
I20260807 03:09:33.489470 26029 raft_consensus.cc:2781] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:33.489905 26029 raft_consensus.cc:686] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 LEADER]: Becoming Leader. State: Replica: c6c7a4bb219f4c109247f187f2cc2e04, State: Running, Role: LEADER
I20260807 03:09:33.490358 26029 consensus_queue.cc:237] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.493343 25468 catalog_manager.cc:5495] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6c7a4bb219f4c109247f187f2cc2e04 (127.17.31.2). New cstate: current_term: 1 leader_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } health_report { overall_health: UNKNOWN } } }
I20260807 03:09:33.526069 26127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:33.520583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.17.31.2:40519)
0807 03:09:33.521098 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:33.521104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:33.521119 (+ 15us) server_negotiation.cc:407] Connection header received
0807 03:09:33.521210 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:33.521213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:33.521289 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:33.521458 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:09:33.521859 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.522650 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.523940 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.524351 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.524479 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:33.524905 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:09:33.525478 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:09:33.525634 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:33.525707 (+ 73us) server_negotiation.cc:299] Negotiation successful
0807 03:09:33.525781 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":250,"threads_started":1}
I20260807 03:09:33.529462 25644 raft_consensus.cc:1261] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Refusing update from remote peer c6c7a4bb219f4c109247f187f2cc2e04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:09:33.529462 25948 raft_consensus.cc:1261] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123 [term 1 FOLLOWER]: Refusing update from remote peer c6c7a4bb219f4c109247f187f2cc2e04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:09:33.530169 26123 consensus_queue.cc:1035] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:33.530568 26029 consensus_queue.cc:1035] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:33.547475 25467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:09:33.547732 25467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:09:33.550516 25467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb: Sending DeleteTablet for 3 replicas of tablet 62f6de515bd243a1a0fabcecb2af272a
I20260807 03:09:33.551266 25618 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f6de515bd243a1a0fabcecb2af272a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:33 UTC) from {username='oryx', principal='oryx/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:38252
I20260807 03:09:33.551335 25773 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f6de515bd243a1a0fabcecb2af272a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:33 UTC) from {username='oryx', principal='oryx/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:43488
I20260807 03:09:33.551546 26100 tablet_replica.cc:330] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035: stopping tablet replica
I20260807 03:09:33.551782 26100 raft_consensus.cc:2229] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:33.551997 26100 raft_consensus.cc:2258] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:33.552276 26101 tablet_replica.cc:330] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04: stopping tablet replica
I20260807 03:09:33.552480 26101 raft_consensus.cc:2229] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:09:33.552929 26101 raft_consensus.cc:2258] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:33.552981 26100 ts_tablet_manager.cc:1905] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:33.553637 25922 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f6de515bd243a1a0fabcecb2af272a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:33 UTC) from {username='oryx', principal='oryx/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:44358
I20260807 03:09:33.553889 26102 tablet_replica.cc:330] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123: stopping tablet replica
I20260807 03:09:33.554064 26102 raft_consensus.cc:2229] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:33.554252 26102 raft_consensus.cc:2258] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:33.554693 25467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:09:33.555382 25467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:09:33.557006 26101 ts_tablet_manager.cc:1905] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:33.559463 26100 ts_tablet_manager.cc:1918] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:09:33.559697 26100 log.cc:1196] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/62f6de515bd243a1a0fabcecb2af272a
I20260807 03:09:33.560189 26100 ts_tablet_manager.cc:1939] T 62f6de515bd243a1a0fabcecb2af272a P 516efccf09904a4c9fb3b54576bfd035: Deleting consensus metadata
I20260807 03:09:33.561255 25452 catalog_manager.cc:4841] TS 516efccf09904a4c9fb3b54576bfd035 (127.17.31.1:37791): tablet 62f6de515bd243a1a0fabcecb2af272a (table test-table [id=77dba33d1c104c56852029a5ed3d406c]) successfully deleted
I20260807 03:09:33.562390 25618 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5ba2eeca6d46bab72d29f6673d718b (DEFAULT_TABLE table=test-table [id=2fbd63ea3ff846609977ded8da353e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:33.562456 25922 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5ba2eeca6d46bab72d29f6673d718b (DEFAULT_TABLE table=test-table [id=2fbd63ea3ff846609977ded8da353e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:33.562659 25773 tablet_service.cc:1467] Processing CreateTablet for tablet 1c5ba2eeca6d46bab72d29f6673d718b (DEFAULT_TABLE table=test-table [id=2fbd63ea3ff846609977ded8da353e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:09:33.562806 25618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5ba2eeca6d46bab72d29f6673d718b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:33.562857 25922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5ba2eeca6d46bab72d29f6673d718b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:33.563056 25773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c5ba2eeca6d46bab72d29f6673d718b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:33.567979 26102 ts_tablet_manager.cc:1905] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:33.569368 26117 tablet_bootstrap.cc:492] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123: Bootstrap starting.
I20260807 03:09:33.571158 26117 tablet_bootstrap.cc:654] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:33.573580 26118 tablet_bootstrap.cc:492] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035: Bootstrap starting.
I20260807 03:09:33.575201 26118 tablet_bootstrap.cc:654] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:33.577764 26120 tablet_bootstrap.cc:492] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04: Bootstrap starting.
I20260807 03:09:33.578644 26102 ts_tablet_manager.cc:1918] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:09:33.578819 26102 log.cc:1196] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/62f6de515bd243a1a0fabcecb2af272a
I20260807 03:09:33.579331 26102 ts_tablet_manager.cc:1939] T 62f6de515bd243a1a0fabcecb2af272a P 247d9927b6344d729e0ddfacda327123: Deleting consensus metadata
I20260807 03:09:33.579470 26120 tablet_bootstrap.cc:654] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:33.580534 25453 catalog_manager.cc:4841] TS 247d9927b6344d729e0ddfacda327123 (127.17.31.3:33165): tablet 62f6de515bd243a1a0fabcecb2af272a (table test-table [id=77dba33d1c104c56852029a5ed3d406c]) successfully deleted
I20260807 03:09:33.581734 26117 tablet_bootstrap.cc:492] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123: No bootstrap required, opened a new log
I20260807 03:09:33.581914 26117 ts_tablet_manager.cc:1397] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20260807 03:09:33.582623 26117 raft_consensus.cc:348] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.582850 26117 raft_consensus.cc:374] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:33.582943 26117 raft_consensus.cc:729] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247d9927b6344d729e0ddfacda327123, State: Initialized, Role: FOLLOWER
I20260807 03:09:33.583178 26117 consensus_queue.cc:260] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.583745 26117 ts_tablet_manager.cc:1428] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260807 03:09:33.589063 26101 ts_tablet_manager.cc:1918] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:09:33.589216 26101 log.cc:1196] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/62f6de515bd243a1a0fabcecb2af272a
I20260807 03:09:33.589648 26101 ts_tablet_manager.cc:1939] T 62f6de515bd243a1a0fabcecb2af272a P c6c7a4bb219f4c109247f187f2cc2e04: Deleting consensus metadata
I20260807 03:09:33.590978 25452 catalog_manager.cc:4841] TS c6c7a4bb219f4c109247f187f2cc2e04 (127.17.31.2:40519): tablet 62f6de515bd243a1a0fabcecb2af272a (table test-table [id=77dba33d1c104c56852029a5ed3d406c]) successfully deleted
I20260807 03:09:33.591676 26120 tablet_bootstrap.cc:492] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04: No bootstrap required, opened a new log
I20260807 03:09:33.591851 26120 ts_tablet_manager.cc:1397] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20260807 03:09:33.592622 26120 raft_consensus.cc:348] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.592867 26120 raft_consensus.cc:374] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:33.592957 26120 raft_consensus.cc:729] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c7a4bb219f4c109247f187f2cc2e04, State: Initialized, Role: FOLLOWER
I20260807 03:09:33.593196 26120 consensus_queue.cc:260] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.593791 26120 ts_tablet_manager.cc:1428] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260807 03:09:33.594969 26118 tablet_bootstrap.cc:492] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035: No bootstrap required, opened a new log
I20260807 03:09:33.595139 26118 ts_tablet_manager.cc:1397] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.004s
I20260807 03:09:33.595899 26118 raft_consensus.cc:348] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.596153 26118 raft_consensus.cc:374] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:33.596237 26118 raft_consensus.cc:729] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516efccf09904a4c9fb3b54576bfd035, State: Initialized, Role: FOLLOWER
I20260807 03:09:33.596494 26118 consensus_queue.cc:260] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.597028 26118 ts_tablet_manager.cc:1428] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260807 03:09:33.883267 26096 raft_consensus.cc:482] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:09:33.883512 26096 raft_consensus.cc:504] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.884272 26096 leader_election.cc:288] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516efccf09904a4c9fb3b54576bfd035 (127.17.31.1:37791), c6c7a4bb219f4c109247f187f2cc2e04 (127.17.31.2:40519)
I20260807 03:09:33.884833 25644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5ba2eeca6d46bab72d29f6673d718b" candidate_uuid: "247d9927b6344d729e0ddfacda327123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516efccf09904a4c9fb3b54576bfd035" is_pre_election: true
I20260807 03:09:33.885076 25644 raft_consensus.cc:2454] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 247d9927b6344d729e0ddfacda327123 in term 0.
I20260807 03:09:33.885527 25871 leader_election.cc:302] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247d9927b6344d729e0ddfacda327123, 516efccf09904a4c9fb3b54576bfd035; no voters:
I20260807 03:09:33.885525 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5ba2eeca6d46bab72d29f6673d718b" candidate_uuid: "247d9927b6344d729e0ddfacda327123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" is_pre_election: true
I20260807 03:09:33.885846 25797 raft_consensus.cc:2454] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 247d9927b6344d729e0ddfacda327123 in term 0.
I20260807 03:09:33.886168 26096 raft_consensus.cc:2781] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:09:33.886292 26096 raft_consensus.cc:482] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:09:33.886406 26096 raft_consensus.cc:3037] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:33.888310 26096 raft_consensus.cc:504] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.888808 26096 leader_election.cc:288] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [CANDIDATE]: Term 1 election: Requested vote from peers 516efccf09904a4c9fb3b54576bfd035 (127.17.31.1:37791), c6c7a4bb219f4c109247f187f2cc2e04 (127.17.31.2:40519)
I20260807 03:09:33.889273 25644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5ba2eeca6d46bab72d29f6673d718b" candidate_uuid: "247d9927b6344d729e0ddfacda327123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516efccf09904a4c9fb3b54576bfd035"
I20260807 03:09:33.889389 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c5ba2eeca6d46bab72d29f6673d718b" candidate_uuid: "247d9927b6344d729e0ddfacda327123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6c7a4bb219f4c109247f187f2cc2e04"
I20260807 03:09:33.889490 25644 raft_consensus.cc:3037] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:33.889585 25797 raft_consensus.cc:3037] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:33.892561 25644 raft_consensus.cc:2454] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 247d9927b6344d729e0ddfacda327123 in term 1.
I20260807 03:09:33.893007 25797 raft_consensus.cc:2454] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 247d9927b6344d729e0ddfacda327123 in term 1.
I20260807 03:09:33.892985 25871 leader_election.cc:302] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247d9927b6344d729e0ddfacda327123, 516efccf09904a4c9fb3b54576bfd035; no voters:
I20260807 03:09:33.893287 26096 raft_consensus.cc:2781] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:33.893671 26096 raft_consensus.cc:686] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [term 1 LEADER]: Becoming Leader. State: Replica: 247d9927b6344d729e0ddfacda327123, State: Running, Role: LEADER
I20260807 03:09:33.894025 26096 consensus_queue.cc:237] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } }
I20260807 03:09:33.896768 25469 catalog_manager.cc:5495] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 reported cstate change: term changed from 0 to 1, leader changed from <none> to 247d9927b6344d729e0ddfacda327123 (127.17.31.3). New cstate: current_term: 1 leader_uuid: "247d9927b6344d729e0ddfacda327123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "247d9927b6344d729e0ddfacda327123" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 33165 } health_report { overall_health: HEALTHY } } }
I20260807 03:09:33.958590 26137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:33.948281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.17.31.3:33165)
0807 03:09:33.952777 (+ 4496us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:33.952784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:33.952801 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:09:33.952864 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:33.952869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:33.952944 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:33.953047 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:09:33.953506 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.954626 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.956547 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:33.957190 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:33.957301 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:33.957780 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:09:33.957913 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:09:33.957993 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:33.958168 (+ 175us) server_negotiation.cc:299] Negotiation successful
0807 03:09:33.958245 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4376,"thread_start_us":4257,"threads_started":1}
I20260807 03:09:33.967850 25797 raft_consensus.cc:1261] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Refusing update from remote peer 247d9927b6344d729e0ddfacda327123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:09:33.967900 25644 raft_consensus.cc:1261] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Refusing update from remote peer 247d9927b6344d729e0ddfacda327123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:09:33.968384 26096 consensus_queue.cc:1035] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6c7a4bb219f4c109247f187f2cc2e04" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:33.968693 26135 consensus_queue.cc:1035] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516efccf09904a4c9fb3b54576bfd035" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 37791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:09:34.015894 25467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:09:34.016208 25467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:09:34.019040 25467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd16109787144a5d8e7bb7550b8a70bb: Sending DeleteTablet for 3 replicas of tablet 1c5ba2eeca6d46bab72d29f6673d718b
I20260807 03:09:34.019670 25618 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5ba2eeca6d46bab72d29f6673d718b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:34 UTC) from {username='oryx', principal='oryx/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:38252
I20260807 03:09:34.019789 25773 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5ba2eeca6d46bab72d29f6673d718b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:34 UTC) from {username='oryx', principal='oryx/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:43488
I20260807 03:09:34.019966 26100 tablet_replica.cc:330] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035: stopping tablet replica
I20260807 03:09:34.019984 25922 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c5ba2eeca6d46bab72d29f6673d718b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:09:34 UTC) from {username='oryx', principal='oryx/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:44358
I20260807 03:09:34.020042 26101 tablet_replica.cc:330] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04: stopping tablet replica
I20260807 03:09:34.020156 26100 raft_consensus.cc:2229] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:34.020198 26101 raft_consensus.cc:2229] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:09:34.020231 26102 tablet_replica.cc:330] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123: stopping tablet replica
I20260807 03:09:34.020354 26100 raft_consensus.cc:2258] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:34.020393 26102 raft_consensus.cc:2229] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:09:34.020427 26101 raft_consensus.cc:2258] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:34.020697 26102 raft_consensus.cc:2258] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:09:34.021490 26100 ts_tablet_manager.cc:1905] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:34.021586 26101 ts_tablet_manager.cc:1905] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:34.021611 26102 ts_tablet_manager.cc:1905] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:09:34.026044 26100 ts_tablet_manager.cc:1918] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 03:09:34.026191 26100 log.cc:1196] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/1c5ba2eeca6d46bab72d29f6673d718b
I20260807 03:09:34.026647 26100 ts_tablet_manager.cc:1939] T 1c5ba2eeca6d46bab72d29f6673d718b P 516efccf09904a4c9fb3b54576bfd035: Deleting consensus metadata
I20260807 03:09:34.027709 25452 catalog_manager.cc:4841] TS 516efccf09904a4c9fb3b54576bfd035 (127.17.31.1:37791): tablet 1c5ba2eeca6d46bab72d29f6673d718b (table test-table [id=2fbd63ea3ff846609977ded8da353e67]) successfully deleted
I20260807 03:09:34.028117 26102 ts_tablet_manager.cc:1918] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 03:09:34.028167 26101 ts_tablet_manager.cc:1918] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 03:09:34.028247 26102 log.cc:1196] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/1c5ba2eeca6d46bab72d29f6673d718b
I20260807 03:09:34.028276 26101 log.cc:1196] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/1c5ba2eeca6d46bab72d29f6673d718b
I20260807 03:09:34.028335 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 25544
I20260807 03:09:34.028695 26102 ts_tablet_manager.cc:1939] T 1c5ba2eeca6d46bab72d29f6673d718b P 247d9927b6344d729e0ddfacda327123: Deleting consensus metadata
I20260807 03:09:34.028695 26101 ts_tablet_manager.cc:1939] T 1c5ba2eeca6d46bab72d29f6673d718b P c6c7a4bb219f4c109247f187f2cc2e04: Deleting consensus metadata
I20260807 03:09:34.029634 25453 catalog_manager.cc:4841] TS 247d9927b6344d729e0ddfacda327123 (127.17.31.3:33165): tablet 1c5ba2eeca6d46bab72d29f6673d718b (table test-table [id=2fbd63ea3ff846609977ded8da353e67]) successfully deleted
I20260807 03:09:34.030133 25452 catalog_manager.cc:4841] TS c6c7a4bb219f4c109247f187f2cc2e04 (127.17.31.2:40519): tablet 1c5ba2eeca6d46bab72d29f6673d718b (table test-table [id=2fbd63ea3ff846609977ded8da353e67]) successfully deleted
Aug 07 03:09:34 dist-test-slave-tm1g krb5kdc[25420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.31.1@KRBTEST.COM
I20260807 03:09:34.045276 26149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:34.029575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38270 (local address 127.17.31.1:37791)
0807 03:09:34.031778 (+ 2203us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:34.031782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:34.031796 (+ 14us) server_negotiation.cc:407] Connection header received
0807 03:09:34.031838 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:34.031841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:34.031893 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:34.032010 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:34.033162 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:34.033894 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:34.035171 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:34.035584 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:34.035720 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:34.040034 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:34.040054 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:34.040078 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:34.040123 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:34.043041 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:34.043647 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:34.043655 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:34.043657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:34.043785 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:34.044100 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:34.044104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:34.044106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:34.044427 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:34.044623 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:34.044857 (+ 234us) server_negotiation.cc:299] Negotiation successful
0807 03:09:34.045021 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2099,"thread_start_us":1996,"threads_started":1}
I20260807 03:09:34.045900 25690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:34.171396 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 25544
I20260807 03:09:34.195149 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 25700
Aug 07 03:09:34 dist-test-slave-tm1g krb5kdc[25420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.31.2@KRBTEST.COM
I20260807 03:09:34.209370 26151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:34.195891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.17.31.2:40519)
0807 03:09:34.196350 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:34.196354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:34.196371 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:09:34.196425 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:34.196429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:34.196486 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:34.196576 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:34.197782 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:34.198568 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:34.199744 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:34.200148 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:34.200240 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:34.204168 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:34.204181 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:34.204197 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:34.204228 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:34.207094 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:34.207660 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:34.207668 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:34.207670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:34.207792 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:34.208093 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:34.208096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:34.208098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:34.208413 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:34.208625 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:34.208829 (+ 204us) server_negotiation.cc:299] Negotiation successful
0807 03:09:34.209004 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":227,"threads_started":1}
I20260807 03:09:34.209985 25844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:34.338433 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 25700
I20260807 03:09:34.365260 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 25853
Aug 07 03:09:34 dist-test-slave-tm1g krb5kdc[25420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.31.3@KRBTEST.COM
I20260807 03:09:34.380390 26137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:34.366014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.17.31.3:33165)
0807 03:09:34.366214 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:34.366222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:34.366243 (+ 21us) server_negotiation.cc:407] Connection header received
0807 03:09:34.366492 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:34.366497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:34.366574 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:34.366679 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:34.368025 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:34.368740 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:34.370032 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:34.370515 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:34.370644 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:34.374951 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:34.374967 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:34.374983 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:34.375032 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:34.377668 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:34.378324 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:34.378335 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:34.378339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:34.378491 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:34.378884 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:34.378889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:34.378891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:34.379240 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:34.379565 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:34.379852 (+ 287us) server_negotiation.cc:299] Negotiation successful
0807 03:09:34.380095 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 03:09:34.381047 25990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:34.504972 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 25853
I20260807 03:09:34.527901 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 25433
I20260807 03:09:34.529055 25512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:34.645170 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 25433
2026-08-07T03:09:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6067 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:09:34 dist-test-slave-tm1g krb5kdc[26159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:09:34 dist-test-slave-tm1g krb5kdc[26159](info): set up 2 sockets
Aug 07 03:09:34 dist-test-slave-tm1g krb5kdc[26159](info): commencing operation
krb5kdc: starting...
W20260807 03:09:35.775816 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:09:35 dist-test-slave-tm1g krb5kdc[26159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:09:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:09:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:09:36.011013 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:32845
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:33041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:32845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:36.335008 26173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:36.335276 26173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:36.335361 26173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:36.342186 26173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:09:36.342279 26173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:36.342365 26173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:36.342442 26173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:09:36.342491 26173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:09:36.351951 26173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:33041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:32845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:32845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:36.353051 26173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:36.354220 26173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:36.365554 26181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:36.365628 26178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:36.365597 26179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:36.366688 26173 server_base.cc:1034] running on GCE node
I20260807 03:09:36.367416 26173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:36.368762 26173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:36.369993 26173 hybrid_clock.cc:648] HybridClock initialized: now 1786072176369960 us; error 38 us; skew 500 ppm
I20260807 03:09:36.370421 26173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-07T03:09:36Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1789 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:09:36 dist-test-slave-tm1g krb5kdc[26187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:09:36 dist-test-slave-tm1g krb5kdc[26187](info): set up 2 sockets
Aug 07 03:09:36 dist-test-slave-tm1g krb5kdc[26187](info): commencing operation
krb5kdc: starting...
W20260807 03:09:37.548627 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:09:37 dist-test-slave-tm1g krb5kdc[26187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:09:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:09:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:09:37.764035 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:45009
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:45009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:38.085417 26200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:38.085682 26200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:38.085835 26200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:38.093075 26200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:09:38.093169 26200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:38.093214 26200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:38.093297 26200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:09:38.093385 26200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:09:38.102893 26200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:45009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:45009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:38.104167 26200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:38.105435 26200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:38.116797 26210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:38.116869 26207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:38.117105 26205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:38.117966 26200 server_base.cc:1034] running on GCE node
I20260807 03:09:38.118710 26200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:38.119979 26200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:38.121172 26200 hybrid_clock.cc:648] HybridClock initialized: now 1786072178121134 us; error 31 us; skew 500 ppm
I20260807 03:09:38.121443 26200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:38 dist-test-slave-tm1g krb5kdc[26187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072178, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:38.125146 26200 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:09:38.126634 26200 webserver.cc:460] Webserver started at http://127.17.31.62:43981/ using document root <none> and password file <none>
I20260807 03:09:38.127327 26200 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:38.127491 26200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:38.127794 26200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:38.130718 26200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "c99a36b794844aa7b8120397d8c35215"
format_stamp: "Formatted at 2026-08-07 03:09:38 on dist-test-slave-tm1g"
server_key: "392e6bdefe4cf834fa7b6dfc5e82cd0e"
server_key_iv: "eb6902bdeb3b46a1657262eb4c69275d"
server_key_version: "encryptionkey@0"
I20260807 03:09:38.131459 26200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "c99a36b794844aa7b8120397d8c35215"
format_stamp: "Formatted at 2026-08-07 03:09:38 on dist-test-slave-tm1g"
server_key: "392e6bdefe4cf834fa7b6dfc5e82cd0e"
server_key_iv: "eb6902bdeb3b46a1657262eb4c69275d"
server_key_version: "encryptionkey@0"
I20260807 03:09:38.136431 26200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 03:09:38.139731 26219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:38.140969 26200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 03:09:38.141148 26200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "c99a36b794844aa7b8120397d8c35215"
format_stamp: "Formatted at 2026-08-07 03:09:38 on dist-test-slave-tm1g"
server_key: "392e6bdefe4cf834fa7b6dfc5e82cd0e"
server_key_iv: "eb6902bdeb3b46a1657262eb4c69275d"
server_key_version: "encryptionkey@0"
I20260807 03:09:38.141359 26200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:38.162544 26200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:38.166409 26200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:38.166687 26200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:38.185092 26200 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:45009
I20260807 03:09:38.185122 26272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:45009 every 8 connection(s)
I20260807 03:09:38.186477 26200 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:09:38.190038 26273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:38.194029 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 26200
I20260807 03:09:38.194274 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:09:38.195125 17532 external_mini_cluster.cc:1428] Setting key 130441f4d466d21ed05147d674a8e724
I20260807 03:09:38.201730 26273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215: Bootstrap starting.
I20260807 03:09:38.205312 26273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:38.206516 26273 log.cc:824] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:38.209108 26273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215: No bootstrap required, opened a new log
Aug 07 03:09:38 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:38.218488 26276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:38.197235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55038 (local address 127.17.31.62:45009)
0807 03:09:38.198016 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:38.198028 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:38.198068 (+ 40us) server_negotiation.cc:407] Connection header received
0807 03:09:38.200288 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:38.200309 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:38.200667 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:38.200963 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:38.202865 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:38.204155 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:38.205974 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:38.206778 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:38.206901 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:38.211356 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:38.211362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:38.211376 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:38.211412 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:38.214550 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:38.215335 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:38.215341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:38.215346 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:38.215492 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:38.215879 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:38.215881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:38.215883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:38.216377 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:38.216598 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:38.217075 (+ 477us) server_negotiation.cc:299] Negotiation successful
0807 03:09:38.217437 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":384,"threads_started":1}
I20260807 03:09:38.218505 26273 raft_consensus.cc:348] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a36b794844aa7b8120397d8c35215" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45009 } }
I20260807 03:09:38.218976 26273 raft_consensus.cc:374] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:38.219091 26273 raft_consensus.cc:729] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99a36b794844aa7b8120397d8c35215, State: Initialized, Role: FOLLOWER
I20260807 03:09:38.219869 26273 consensus_queue.cc:260] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a36b794844aa7b8120397d8c35215" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45009 } }
I20260807 03:09:38.220191 26278 raft_consensus.cc:482] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:09:38.220479 26278 raft_consensus.cc:504] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a36b794844aa7b8120397d8c35215" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45009 } }
I20260807 03:09:38.221004 26273 raft_consensus.cc:388] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:09:38.221172 26273 raft_consensus.cc:482] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:09:38.221005 26278 leader_election.cc:302] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c99a36b794844aa7b8120397d8c35215; no voters:
I20260807 03:09:38.221298 26273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:38.222131 26278 leader_election.cc:288] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260807 03:09:38.223724 26273 raft_consensus.cc:504] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a36b794844aa7b8120397d8c35215" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45009 } }
I20260807 03:09:38.223974 26273 leader_election.cc:302] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c99a36b794844aa7b8120397d8c35215; no voters:
I20260807 03:09:38.224054 26279 raft_consensus.cc:2741] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260807 03:09:38.224195 26273 leader_election.cc:288] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:09:38.224535 26278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:38.225746 26278 raft_consensus.cc:686] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 1 LEADER]: Becoming Leader. State: Replica: c99a36b794844aa7b8120397d8c35215, State: Running, Role: LEADER
I20260807 03:09:38.226049 26273 sys_catalog.cc:564] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:09:38.226284 26278 consensus_queue.cc:237] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a36b794844aa7b8120397d8c35215" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45009 } }
I20260807 03:09:38.228345 26279 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c99a36b794844aa7b8120397d8c35215. Latest consensus state: current_term: 1 leader_uuid: "c99a36b794844aa7b8120397d8c35215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a36b794844aa7b8120397d8c35215" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45009 } } }
I20260807 03:09:38.228295 26280 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c99a36b794844aa7b8120397d8c35215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a36b794844aa7b8120397d8c35215" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45009 } } }
I20260807 03:09:38.228734 26279 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:38.228783 26280 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:38.229566 26284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:09:38.234637 26284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:09:38.247913 26284 catalog_manager.cc:1347] Generated new cluster ID: 2e5cbbfb764542348ef6980a184d0494
I20260807 03:09:38.248028 26284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:09:38.280573 26284 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:09:38.281869 26284 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:09:38.291802 26284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215: Generated new TSK 0
I20260807 03:09:38.292603 26284 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:09:38.351321 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:45009
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:38.680814 26301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:38.681207 26301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:38.681385 26301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:38.688675 26301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:38.688827 26301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:38.688999 26301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:09:38.698634 26301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:45009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:38.699975 26301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:38.701292 26301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:38.713302 26307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:38.714676 26308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:38.715085 26310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:38.715849 26301 server_base.cc:1034] running on GCE node
I20260807 03:09:38.716468 26301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:38.717346 26301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:38.718571 26301 hybrid_clock.cc:648] HybridClock initialized: now 1786072178718536 us; error 71 us; skew 500 ppm
I20260807 03:09:38.718878 26301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:38 dist-test-slave-tm1g krb5kdc[26187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072178, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:38.725565 26301 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:09:38.727176 26301 webserver.cc:460] Webserver started at http://127.17.31.1:37979/ using document root <none> and password file <none>
I20260807 03:09:38.727871 26301 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:38.728026 26301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:38.728341 26301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:38.731278 26301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "9b2eb42b5f904f84bc266925ce11bcd5"
format_stamp: "Formatted at 2026-08-07 03:09:38 on dist-test-slave-tm1g"
server_key: "41f7ba0d3b5d16fa4536bd6c59759ccd"
server_key_iv: "6ad8a75d3d46b777335d8cc8a6b2b974"
server_key_version: "encryptionkey@0"
I20260807 03:09:38.732085 26301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "9b2eb42b5f904f84bc266925ce11bcd5"
format_stamp: "Formatted at 2026-08-07 03:09:38 on dist-test-slave-tm1g"
server_key: "41f7ba0d3b5d16fa4536bd6c59759ccd"
server_key_iv: "6ad8a75d3d46b777335d8cc8a6b2b974"
server_key_version: "encryptionkey@0"
I20260807 03:09:38.737375 26301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:09:38.740916 26317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:38.742203 26301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:09:38.742405 26301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "9b2eb42b5f904f84bc266925ce11bcd5"
format_stamp: "Formatted at 2026-08-07 03:09:38 on dist-test-slave-tm1g"
server_key: "41f7ba0d3b5d16fa4536bd6c59759ccd"
server_key_iv: "6ad8a75d3d46b777335d8cc8a6b2b974"
server_key_version: "encryptionkey@0"
I20260807 03:09:38.742604 26301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:38.758292 26301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:38.762001 26301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:38.762364 26301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:38.763304 26301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:38.764603 26301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:38.764706 26301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:38.764869 26301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:38.764945 26301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:38.797919 26301 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:38549
I20260807 03:09:38.797974 26437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:38549 every 8 connection(s)
I20260807 03:09:38.799292 26301 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:09:38.803443 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 26301
I20260807 03:09:38.803675 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:09:38.804518 17532 external_mini_cluster.cc:1428] Setting key 6bdd902711773cd06f1c9746735fb6e7
Aug 07 03:09:38 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072178, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:38.821766 26440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:38.801740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:60385 (local address 127.17.31.62:45009)
0807 03:09:38.802191 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:38.802197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:38.803392 (+ 1195us) server_negotiation.cc:407] Connection header received
0807 03:09:38.805908 (+ 2516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:38.805913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:38.806016 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:38.806134 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:38.808123 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:38.808861 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:38.811354 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:38.811757 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:38.811859 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:38.816094 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:38.816099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:38.816102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:38.816134 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:38.818710 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:38.819452 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:38.819457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:38.819459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:38.819574 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:38.820009 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:38.820027 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:38.820030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:38.820290 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:38.820448 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:38.821162 (+ 714us) server_negotiation.cc:299] Negotiation successful
0807 03:09:38.821394 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":212,"threads_started":1}
I20260807 03:09:38.822803 26441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:38.802405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:60385)
0807 03:09:38.803194 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:38.803254 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:38.805644 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:38.806413 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:38.806421 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:38.807123 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:38.807938 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:38.807964 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:38.809025 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:38.809033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:38.811206 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:38.811224 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:38.811994 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:38.812003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:38.812341 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:38.813104 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:38.813139 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:38.815838 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:38.818861 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:38.818870 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:38.818888 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:38.819307 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:38.819688 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:38.819693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:38.819697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:38.819891 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:38.820422 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:38.820430 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:38.820781 (+ 351us) client_negotiation.cc:769] Sending connection context
0807 03:09:38.821045 (+ 264us) client_negotiation.cc:240] Negotiation successful
0807 03:09:38.821589 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":268,"threads_started":1}
I20260807 03:09:38.824479 26438 heartbeater.cc:347] Connected to a master server at 127.17.31.62:45009
I20260807 03:09:38.824780 26438 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:38.825526 26438 heartbeater.cc:511] Master 127.17.31.62:45009 requested a full tablet report, sending...
I20260807 03:09:38.827833 26237 ts_manager.cc:194] Registered new tserver with Master: 9b2eb42b5f904f84bc266925ce11bcd5 (127.17.31.1:38549)
I20260807 03:09:38.829277 26237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:60385
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:09:38.860702 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:45009
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:39.185842 26444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:39.186092 26444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:39.186254 26444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:39.193450 26444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:39.193563 26444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:39.193744 26444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:09:39.203321 26444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:45009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:39.204510 26444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:39.205766 26444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:39.218688 26450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:39.218621 26453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:39.218652 26451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:39.219753 26444 server_base.cc:1034] running on GCE node
I20260807 03:09:39.220330 26444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:39.221238 26444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:39.222471 26444 hybrid_clock.cc:648] HybridClock initialized: now 1786072179222420 us; error 55 us; skew 500 ppm
I20260807 03:09:39.222867 26444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:39 dist-test-slave-tm1g krb5kdc[26187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072179, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:39.225876 26444 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:09:39.227362 26444 webserver.cc:460] Webserver started at http://127.17.31.2:36669/ using document root <none> and password file <none>
I20260807 03:09:39.228073 26444 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:39.228215 26444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:39.228525 26444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:39.231344 26444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "4a59f49164da4e17a62d0a58cb2b68ce"
format_stamp: "Formatted at 2026-08-07 03:09:39 on dist-test-slave-tm1g"
server_key: "af3430d49521e0ad3ced7855ae548d33"
server_key_iv: "53f586578954e2190036e2ced06301c0"
server_key_version: "encryptionkey@0"
I20260807 03:09:39.232125 26444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "4a59f49164da4e17a62d0a58cb2b68ce"
format_stamp: "Formatted at 2026-08-07 03:09:39 on dist-test-slave-tm1g"
server_key: "af3430d49521e0ad3ced7855ae548d33"
server_key_iv: "53f586578954e2190036e2ced06301c0"
server_key_version: "encryptionkey@0"
I20260807 03:09:39.237043 26444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:09:39.240401 26460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:39.241679 26444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260807 03:09:39.241847 26444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "4a59f49164da4e17a62d0a58cb2b68ce"
format_stamp: "Formatted at 2026-08-07 03:09:39 on dist-test-slave-tm1g"
server_key: "af3430d49521e0ad3ced7855ae548d33"
server_key_iv: "53f586578954e2190036e2ced06301c0"
server_key_version: "encryptionkey@0"
I20260807 03:09:39.242030 26444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:39.255962 26444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:39.260149 26444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:39.260428 26444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:39.261364 26444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:39.262673 26444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:39.262744 26444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:39.262908 26444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:39.262964 26444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:39.295594 26444 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:41333
I20260807 03:09:39.295680 26580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:41333 every 8 connection(s)
I20260807 03:09:39.296871 26444 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:09:39.301898 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 26444
I20260807 03:09:39.302114 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:09:39.302627 17532 external_mini_cluster.cc:1428] Setting key 851e1afebf0bca8716c7527f847ea719
Aug 07 03:09:39 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072179, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:39.316524 26440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:39.299459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:45127 (local address 127.17.31.62:45009)
0807 03:09:39.299655 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:39.299660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:39.300818 (+ 1158us) server_negotiation.cc:407] Connection header received
0807 03:09:39.302790 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:39.302795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:39.302904 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:39.303025 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:39.304683 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:39.305402 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:39.307024 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:39.307456 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:39.307593 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:39.310941 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:39.310946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:39.310949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:39.310981 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:39.313498 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:39.314210 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:39.314217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:39.314225 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:39.314412 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:39.314849 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:39.314853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:39.314857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:39.315118 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:39.315268 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:39.315936 (+ 668us) server_negotiation.cc:299] Negotiation successful
0807 03:09:39.316172 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 03:09:39.317440 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:39.299943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:45127)
0807 03:09:39.300661 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:39.300698 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:39.302525 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:39.303249 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:39.303257 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:39.303881 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:39.304527 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:39.304545 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:39.305550 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:39.305555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:39.306872 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:39.306884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:39.307688 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:39.307695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:39.307905 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:39.308580 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:39.308600 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:39.310726 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:39.313659 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:39.313667 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:39.313682 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:39.314084 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:39.314537 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:39.314542 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:39.314545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:39.314728 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:39.315254 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:39.315261 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:39.315577 (+ 316us) client_negotiation.cc:769] Sending connection context
0807 03:09:39.315832 (+ 255us) client_negotiation.cc:240] Negotiation successful
0807 03:09:39.316344 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":302,"threads_started":1}
I20260807 03:09:39.318895 26581 heartbeater.cc:347] Connected to a master server at 127.17.31.62:45009
I20260807 03:09:39.319212 26581 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:39.319905 26581 heartbeater.cc:511] Master 127.17.31.62:45009 requested a full tablet report, sending...
I20260807 03:09:39.321400 26237 ts_manager.cc:194] Registered new tserver with Master: 4a59f49164da4e17a62d0a58cb2b68ce (127.17.31.2:41333)
I20260807 03:09:39.322676 26237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:45127
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:09:39.357594 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:45009
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:39.679404 26586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:39.679678 26586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:39.679821 26586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:39.686884 26586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:39.686978 26586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:39.687081 26586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:09:39.696295 26586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:45009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:39.697516 26586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:39.698837 26586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:39.711422 26591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:39.711470 26594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:39.711500 26592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:39.712647 26586 server_base.cc:1034] running on GCE node
I20260807 03:09:39.713157 26586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:39.713933 26586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:39.715142 26586 hybrid_clock.cc:648] HybridClock initialized: now 1786072179715092 us; error 46 us; skew 500 ppm
I20260807 03:09:39.715400 26586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:39 dist-test-slave-tm1g krb5kdc[26187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072179, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:39.718714 26586 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:09:39.720875 26586 webserver.cc:460] Webserver started at http://127.17.31.3:44087/ using document root <none> and password file <none>
I20260807 03:09:39.721539 26586 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:39.721683 26586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:39.722013 26586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:39.724975 26586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "427b4da3f6ea4d608e4bbc17cc0629b5"
format_stamp: "Formatted at 2026-08-07 03:09:39 on dist-test-slave-tm1g"
server_key: "df1a22e677fdec3e82999da234f2f9df"
server_key_iv: "7a669ea5a1902b6efe5e3f03e80c3540"
server_key_version: "encryptionkey@0"
I20260807 03:09:39.725766 26586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "427b4da3f6ea4d608e4bbc17cc0629b5"
format_stamp: "Formatted at 2026-08-07 03:09:39 on dist-test-slave-tm1g"
server_key: "df1a22e677fdec3e82999da234f2f9df"
server_key_iv: "7a669ea5a1902b6efe5e3f03e80c3540"
server_key_version: "encryptionkey@0"
I20260807 03:09:39.731063 26586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 03:09:39.734391 26601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:39.735638 26586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:09:39.735823 26586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "427b4da3f6ea4d608e4bbc17cc0629b5"
format_stamp: "Formatted at 2026-08-07 03:09:39 on dist-test-slave-tm1g"
server_key: "df1a22e677fdec3e82999da234f2f9df"
server_key_iv: "7a669ea5a1902b6efe5e3f03e80c3540"
server_key_version: "encryptionkey@0"
I20260807 03:09:39.736029 26586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:39.771611 26586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:39.775476 26586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:39.775766 26586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:39.776714 26586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:39.778007 26586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:39.778110 26586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:39.778261 26586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:39.778366 26586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260807 03:09:39.811818 26586 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:45105
I20260807 03:09:39.811872 26718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:45105 every 8 connection(s)
I20260807 03:09:39.813035 26586 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:09:39.819326 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 26586
I20260807 03:09:39.819537 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:09:39.820093 17532 external_mini_cluster.cc:1428] Setting key f53008cc5dd7c614a8b3b7881ed8d3f5
Aug 07 03:09:39 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072179, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:09:39.832247 26438 heartbeater.cc:503] Master 127.17.31.62:45009 was elected leader, sending a full tablet report...
I20260807 03:09:39.832681 26440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:39.815620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:59739 (local address 127.17.31.62:45009)
0807 03:09:39.815840 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:39.815855 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:39.816945 (+ 1090us) server_negotiation.cc:407] Connection header received
0807 03:09:39.818839 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:39.818849 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:39.818918 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:39.819010 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:39.820979 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:39.821801 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:39.823359 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:39.823771 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:39.823920 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:39.827300 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:39.827306 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:39.827312 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:39.827365 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:39.829728 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:39.830459 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:39.830471 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:39.830474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:39.830598 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:39.831029 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:39.831032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:39.831034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:39.831284 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:39.831434 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:39.832116 (+ 682us) server_negotiation.cc:299] Negotiation successful
0807 03:09:39.832342 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 03:09:39.833724 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:39.816032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:59739)
0807 03:09:39.816762 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:39.816825 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:39.818598 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:39.819286 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:39.819298 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:39.820029 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:39.820774 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:39.820793 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:39.821969 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:39.821975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:39.823206 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:39.823218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:39.824006 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:39.824015 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:39.824245 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:39.824859 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:39.824880 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:39.827109 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:39.829873 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:39.829881 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:39.829901 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:39.830275 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:39.830711 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:09:39.830717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:09:39.830720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:09:39.830898 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:09:39.831420 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:39.831427 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:39.831730 (+ 303us) client_negotiation.cc:769] Sending connection context
0807 03:09:39.831986 (+ 256us) client_negotiation.cc:240] Negotiation successful
0807 03:09:39.832487 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":343,"threads_started":1}
I20260807 03:09:39.835196 26719 heartbeater.cc:347] Connected to a master server at 127.17.31.62:45009
I20260807 03:09:39.835476 26719 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:39.836113 26719 heartbeater.cc:511] Master 127.17.31.62:45009 requested a full tablet report, sending...
I20260807 03:09:39.837375 26237 ts_manager.cc:194] Registered new tserver with Master: 427b4da3f6ea4d608e4bbc17cc0629b5 (127.17.31.3:45105)
I20260807 03:09:39.838270 26237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:59739
I20260807 03:09:39.847553 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.1.keytab.
Entry for principal oryx/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.1.keytab.
WARNING: no policy specified for oryx/127.17.31.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.2.keytab.
Entry for principal oryx/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.2.keytab.
WARNING: no policy specified for oryx/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.3.keytab.
Entry for principal oryx/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.3.keytab.
I20260807 03:09:39.979054 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 26301
Aug 07 03:09:39 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:09:39.995741 26729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:39.980248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.17.31.1:38549)
0807 03:09:39.980899 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:39.980908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:39.980943 (+ 35us) server_negotiation.cc:407] Connection header received
0807 03:09:39.981029 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:39.981043 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:39.981262 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:39.981445 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:39.982950 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:39.983994 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:39.985354 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:39.985922 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:39.986069 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:39.990188 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:39.990199 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:39.990213 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:39.990249 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:39.992843 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:39.993608 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:39.993616 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:39.993618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:39.993743 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:39.994121 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:39.994127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:39.994130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:39.994531 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:39.994837 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:39.995146 (+ 309us) server_negotiation.cc:299] Negotiation successful
0807 03:09:39.995405 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":291,"threads_started":1}
I20260807 03:09:39.996953 26433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:40.079875 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 26301
I20260807 03:09:40.097566 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 26444
Aug 07 03:09:40 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:09:40.112607 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:40.098471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51994 (local address 127.17.31.2:41333)
0807 03:09:40.099004 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:40.099011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:40.099036 (+ 25us) server_negotiation.cc:407] Connection header received
0807 03:09:40.099097 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:40.099104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:40.099319 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:40.099450 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:40.100622 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:40.101623 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:40.102939 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:40.103514 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:40.103625 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:40.107690 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:40.107700 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:40.107714 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:40.107743 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:40.110244 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:40.110956 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:40.110962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:40.110964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:40.111073 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:40.111375 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:40.111378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:40.111380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:40.111675 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:40.111898 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:40.112122 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 03:09:40.112307 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":270,"threads_started":1}
I20260807 03:09:40.113835 26576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:40.196986 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 26444
I20260807 03:09:40.213678 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 26586
Aug 07 03:09:40 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:09:40.229651 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:40.214759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.17.31.3:45105)
0807 03:09:40.215381 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:40.215387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:40.215415 (+ 28us) server_negotiation.cc:407] Connection header received
0807 03:09:40.215497 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:40.215503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:40.215711 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:40.215846 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:40.217134 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:40.218140 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:40.219507 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:40.220053 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:40.220149 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:40.224356 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:40.224366 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:40.224380 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:40.224412 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:40.227207 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:40.227866 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:40.227872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:40.227875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:40.227995 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:40.228298 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:40.228301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:40.228303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:40.228603 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:40.228966 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:40.229104 (+ 138us) server_negotiation.cc:299] Negotiation successful
0807 03:09:40.229296 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":317,"threads_started":1}
I20260807 03:09:40.231029 26714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:40.314905 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 26586
I20260807 03:09:40.331882 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 26200
I20260807 03:09:40.333139 26268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:40.436586 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 26200
I20260807 03:09:40.458801 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:45009
--webserver_interface=127.17.31.62
--webserver_port=43981
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:45009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:09:40.783253 26736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:40.783537 26736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:40.783713 26736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:40.790665 26736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:09:40.790787 26736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:40.790869 26736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:40.790961 26736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:09:40.791047 26736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:09:40.800457 26736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:45009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:45009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=43981
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:40.801709 26736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:40.802997 26736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:40.814445 26741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:40.814388 26742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:40.814419 26744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:40.815979 26736 server_base.cc:1034] running on GCE node
I20260807 03:09:40.816716 26736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:40.818115 26736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:40.819336 26736 hybrid_clock.cc:648] HybridClock initialized: now 1786072180819266 us; error 75 us; skew 500 ppm
I20260807 03:09:40.819681 26736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:40 dist-test-slave-tm1g krb5kdc[26187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072180, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:40.822674 26736 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:09:40.824122 26736 webserver.cc:460] Webserver started at http://127.17.31.62:43981/ using document root <none> and password file <none>
I20260807 03:09:40.824813 26736 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:40.824965 26736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:40.830266 26736 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:09:40.832885 26752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:40.834158 26736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:09:40.834378 26736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "c99a36b794844aa7b8120397d8c35215"
format_stamp: "Formatted at 2026-08-07 03:09:38 on dist-test-slave-tm1g"
server_key: "392e6bdefe4cf834fa7b6dfc5e82cd0e"
server_key_iv: "eb6902bdeb3b46a1657262eb4c69275d"
server_key_version: "encryptionkey@0"
I20260807 03:09:40.835361 26736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:40.875728 26736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:40.879645 26736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:40.879912 26736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:40.898406 26736 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:45009
I20260807 03:09:40.898432 26805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:45009 every 8 connection(s)
I20260807 03:09:40.899845 26736 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:09:40.901619 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 26736
I20260807 03:09:40.902477 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:38549
--local_ip_for_outbound_sockets=127.17.31.1
--tserver_master_addrs=127.17.31.62:45009
--webserver_port=37979
--webserver_interface=127.17.31.1
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.31.1
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:09:40.906581 26806 sys_catalog.cc:263] Verifying existing consensus state
I20260807 03:09:40.909668 26806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215: Bootstrap starting.
I20260807 03:09:40.928115 26806 log.cc:824] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:40.936746 26806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:09:40.937265 26806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215: Bootstrap complete.
I20260807 03:09:40.946817 26806 raft_consensus.cc:348] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a36b794844aa7b8120397d8c35215" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45009 } }
I20260807 03:09:40.947621 26806 raft_consensus.cc:729] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99a36b794844aa7b8120397d8c35215, State: Initialized, Role: FOLLOWER
I20260807 03:09:40.948221 26806 consensus_queue.cc:260] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a36b794844aa7b8120397d8c35215" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45009 } }
I20260807 03:09:40.948468 26806 raft_consensus.cc:388] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:09:40.948585 26806 raft_consensus.cc:482] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:09:40.948762 26806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:09:40.952473 26806 raft_consensus.cc:504] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a36b794844aa7b8120397d8c35215" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45009 } }
I20260807 03:09:40.953090 26806 leader_election.cc:302] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c99a36b794844aa7b8120397d8c35215; no voters:
I20260807 03:09:40.953852 26806 leader_election.cc:288] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 03:09:40.953994 26810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:09:40.956182 26806 sys_catalog.cc:564] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:09:40.961041 26810 raft_consensus.cc:686] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [term 2 LEADER]: Becoming Leader. State: Replica: c99a36b794844aa7b8120397d8c35215, State: Running, Role: LEADER
I20260807 03:09:40.961699 26810 consensus_queue.cc:237] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a36b794844aa7b8120397d8c35215" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45009 } }
I20260807 03:09:40.970000 26811 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c99a36b794844aa7b8120397d8c35215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a36b794844aa7b8120397d8c35215" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45009 } } }
I20260807 03:09:40.970063 26812 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c99a36b794844aa7b8120397d8c35215. Latest consensus state: current_term: 2 leader_uuid: "c99a36b794844aa7b8120397d8c35215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99a36b794844aa7b8120397d8c35215" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 45009 } } }
I20260807 03:09:40.970470 26811 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:40.971273 26812 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215 [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:40.971532 26825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:09:40.977985 26825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:09:40.979694 26825 catalog_manager.cc:1259] Loaded cluster ID: 2e5cbbfb764542348ef6980a184d0494
I20260807 03:09:40.979818 26825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:09:40.984630 26828 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 03:09:40.984879 26825 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:09:40.986230 26825 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c99a36b794844aa7b8120397d8c35215: Loaded TSK: 0
I20260807 03:09:40.987360 26825 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 03:09:41.264411 26808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:41.264710 26808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:41.264873 26808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:41.271878 26808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:41.272013 26808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:41.272168 26808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:09:41.281599 26808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.1.keytab
--principal=oryx/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:38549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=37979
--tserver_master_addrs=127.17.31.62:45009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:41.282933 26808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:41.284247 26808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:41.297384 26834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:41.297566 26833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:41.297698 26836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:41.298847 26808 server_base.cc:1034] running on GCE node
I20260807 03:09:41.299467 26808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:41.300328 26808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:41.301550 26808 hybrid_clock.cc:648] HybridClock initialized: now 1786072181301497 us; error 47 us; skew 500 ppm
I20260807 03:09:41.301843 26808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:41 dist-test-slave-tm1g krb5kdc[26187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072181, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:41.304895 26808 init.cc:377] Logged in from keytab as oryx/127.17.31.1@KRBTEST.COM (short username oryx)
I20260807 03:09:41.306406 26808 webserver.cc:460] Webserver started at http://127.17.31.1:37979/ using document root <none> and password file <none>
I20260807 03:09:41.307081 26808 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:41.307219 26808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:41.312698 26808 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 03:09:41.315934 26843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:41.317278 26808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 03:09:41.317440 26808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "9b2eb42b5f904f84bc266925ce11bcd5"
format_stamp: "Formatted at 2026-08-07 03:09:38 on dist-test-slave-tm1g"
server_key: "41f7ba0d3b5d16fa4536bd6c59759ccd"
server_key_iv: "6ad8a75d3d46b777335d8cc8a6b2b974"
server_key_version: "encryptionkey@0"
I20260807 03:09:41.318414 26808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:41.333928 26808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:41.338094 26808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:41.338430 26808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:41.339501 26808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:41.340948 26808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:41.341055 26808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:41.341167 26808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:41.341264 26808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:41.378257 26808 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:38549
I20260807 03:09:41.378387 26960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:38549 every 8 connection(s)
I20260807 03:09:41.379702 26808 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:09:41.386011 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 26808
I20260807 03:09:41.386691 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:41333
--local_ip_for_outbound_sockets=127.17.31.2
--tserver_master_addrs=127.17.31.62:45009
--webserver_port=36669
--webserver_interface=127.17.31.2
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.31.2
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 03:09:41 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:41 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:41.399501 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:41.382770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:42627)
0807 03:09:41.383778 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:41.383816 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:41.385663 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:41.387200 (+ 1537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:41.387214 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:41.387999 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:41.388760 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:41.388786 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.390291 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.390299 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:41.392392 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:41.392422 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.393390 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.393399 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:41.393723 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:41.394440 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:41.394472 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:41.398642 (+ 4170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":538,"threads_started":1}
W20260807 03:09:41.400146 26961 heartbeater.cc:644] Failed to heartbeat to 127.17.31.62:45009 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.31.62:45009: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
W20260807 03:09:41.400171 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:41.382753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:42627 (local address 127.17.31.62:45009)
0807 03:09:41.383568 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:41.383585 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:41.383931 (+ 346us) server_negotiation.cc:407] Connection header received
0807 03:09:41.386289 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:41.386343 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:41.386672 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:41.387005 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:41.388943 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.390129 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.392537 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.393142 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.393283 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:41.399047 (+ 5764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:42627: BlockingRecv error: recv got EOF from 127.17.31.1:42627 (error 108)
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":287,"threads_started":1}
Aug 07 03:09:41 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:41 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:41.408056 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:41.400683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:42531)
0807 03:09:41.400888 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:41.400903 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:41.401001 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:41.401335 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:41.401340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:41.401661 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:41.401786 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:41.401793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.402823 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.402832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:41.403685 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:41.403694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.404347 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.404352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:41.404516 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:41.405052 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:41.405067 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:41.407874 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121}
W20260807 03:09:41.408428 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:41.400915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:42531 (local address 127.17.31.62:45009)
0807 03:09:41.401103 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:41.401108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:41.401128 (+ 20us) server_negotiation.cc:407] Connection header received
0807 03:09:41.401164 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:41.401168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:41.401217 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:41.401317 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:41.401917 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.402687 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.403826 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.404216 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.404337 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:41.408155 (+ 3818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:42531: BlockingRecv error: recv got EOF from 127.17.31.1:42531 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 07 03:09:41 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:41 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:41.415953 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:41.408749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:42621)
0807 03:09:41.408958 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:41.408980 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:41.409084 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:41.409379 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:41.409383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:41.409685 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:41.409812 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:41.409819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.410810 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.410815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:41.411687 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:41.411697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.412351 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.412357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:41.412538 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:41.413118 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:41.413131 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:41.415644 (+ 2513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260807 03:09:41.416103 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:41.408944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:42621 (local address 127.17.31.62:45009)
0807 03:09:41.409080 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:41.409084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:41.409100 (+ 16us) server_negotiation.cc:407] Connection header received
0807 03:09:41.409189 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:41.409193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:41.409252 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:41.409361 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:41.409942 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.410676 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.411821 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.412210 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.412330 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:41.415832 (+ 3502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:42621: BlockingRecv error: recv got EOF from 127.17.31.1:42621 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260807 03:09:41.416246 26961 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 03:09:41.710779 26965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:41.711069 26965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:41.711236 26965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:41.718057 26965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:41.718190 26965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:41.718421 26965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:09:41.727756 26965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.2.keytab
--principal=oryx/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:41333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=36669
--tserver_master_addrs=127.17.31.62:45009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:41.728989 26965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:41.730228 26965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:41.742937 26971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:41.743016 26970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:41.751816 26973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:41.753371 26965 server_base.cc:1034] running on GCE node
I20260807 03:09:41.754073 26965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:41.755347 26965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:41.756608 26965 hybrid_clock.cc:648] HybridClock initialized: now 1786072181756528 us; error 49 us; skew 500 ppm
I20260807 03:09:41.756934 26965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:41 dist-test-slave-tm1g krb5kdc[26187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072181, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:41.760560 26965 init.cc:377] Logged in from keytab as oryx/127.17.31.2@KRBTEST.COM (short username oryx)
I20260807 03:09:41.761940 26965 webserver.cc:460] Webserver started at http://127.17.31.2:36669/ using document root <none> and password file <none>
I20260807 03:09:41.762778 26965 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:41.762892 26965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:41.768090 26965 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 03:09:41.770596 26981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:41.771835 26965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:09:41.771966 26965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "4a59f49164da4e17a62d0a58cb2b68ce"
format_stamp: "Formatted at 2026-08-07 03:09:39 on dist-test-slave-tm1g"
server_key: "af3430d49521e0ad3ced7855ae548d33"
server_key_iv: "53f586578954e2190036e2ced06301c0"
server_key_version: "encryptionkey@0"
I20260807 03:09:41.772908 26965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:41.795526 26965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:41.799593 26965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:41.799808 26965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:41.800675 26965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:41.801894 26965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:41.801981 26965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:41.802064 26965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:41.802119 26965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:41.835624 26965 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:41333
I20260807 03:09:41.835708 27099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:41333 every 8 connection(s)
I20260807 03:09:41.837059 26965 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:09:41.844188 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 26965
I20260807 03:09:41.844914 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:45105
--local_ip_for_outbound_sockets=127.17.31.3
--tserver_master_addrs=127.17.31.62:45009
--webserver_port=44087
--webserver_interface=127.17.31.3
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.31.3
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 03:09:41 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:41 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 03:09:41.853045 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:41.839527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:60461 (local address 127.17.31.62:45009)
0807 03:09:41.839776 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:41.839783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:41.840987 (+ 1204us) server_negotiation.cc:407] Connection header received
0807 03:09:41.842825 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:41.842829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:41.842894 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:41.842971 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:41.844471 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.845607 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.847184 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.847899 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.848054 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:41.852734 (+ 4680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:60461: BlockingRecv error: recv got EOF from 127.17.31.2:60461 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260807 03:09:41.853453 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:41.840142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:60461)
0807 03:09:41.840829 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:41.840866 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:41.842581 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:41.843160 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:41.843167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:41.843670 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:41.844221 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:41.844238 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.845762 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.845766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:41.847045 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:41.847057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.848105 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.848110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:41.848293 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:41.848943 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:41.848964 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:41.852480 (+ 3516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":279,"threads_started":1}
W20260807 03:09:41.854226 27100 heartbeater.cc:644] Failed to heartbeat to 127.17.31.62:45009 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.31.62:45009: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Aug 07 03:09:41 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:41 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:41.863546 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:41.854803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:59365)
0807 03:09:41.854968 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:41.854990 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:41.855088 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:41.855357 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:41.855364 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:41.855797 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:41.855998 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:41.856011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.857010 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.857018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:41.858718 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:41.858735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.859390 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.859398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:41.859632 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:41.860189 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:41.860216 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:41.863259 (+ 3043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 03:09:41.863739 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:41.854958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:59365 (local address 127.17.31.62:45009)
0807 03:09:41.855103 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:41.855107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:41.855121 (+ 14us) server_negotiation.cc:407] Connection header received
0807 03:09:41.855180 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:41.855184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:41.855237 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:41.855312 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:41.856140 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.856880 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.858849 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.859237 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.859333 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:41.863473 (+ 4140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:59365: BlockingRecv error: recv got EOF from 127.17.31.2:59365 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 07 03:09:41 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:41 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:41.871953 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:41.864346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:40949)
0807 03:09:41.864497 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:41.864525 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:41.864595 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:41.864927 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:41.864930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:41.865174 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:41.865320 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:41.865328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.866829 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.866832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:41.867713 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:41.867723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.868735 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.868738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:41.868852 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:41.869370 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:41.869387 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:41.871769 (+ 2382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 03:09:41.872283 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:41.864445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:40949 (local address 127.17.31.62:45009)
0807 03:09:41.864636 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:41.864644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:41.864664 (+ 20us) server_negotiation.cc:407] Connection header received
0807 03:09:41.864715 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:41.864722 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:41.864796 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:41.864896 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:41.865455 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.866700 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.867847 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:41.868605 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:41.868752 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:41.872029 (+ 3277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:40949: BlockingRecv error: recv got EOF from 127.17.31.2:40949 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260807 03:09:41.872368 27100 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 03:09:42.180241 27103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:42.180552 27103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:42.180712 27103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:42.187683 27103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:42.187774 27103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:42.187880 27103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:09:42.197113 27103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/krb5kdc/oryx_127.17.31.3.keytab
--principal=oryx/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:45105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=44087
--tserver_master_addrs=127.17.31.62:45009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:42.198355 27103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:42.199559 27103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:42.212067 27109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:42.211998 27112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:42.212514 27108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:42.213245 27103 server_base.cc:1034] running on GCE node
I20260807 03:09:42.213861 27103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:42.214767 27103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:42.215978 27103 hybrid_clock.cc:648] HybridClock initialized: now 1786072182215943 us; error 45 us; skew 500 ppm
I20260807 03:09:42.216269 27103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:09:42 dist-test-slave-tm1g krb5kdc[26187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072182, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:09:42.219358 27103 init.cc:377] Logged in from keytab as oryx/127.17.31.3@KRBTEST.COM (short username oryx)
I20260807 03:09:42.220620 27103 webserver.cc:460] Webserver started at http://127.17.31.3:44087/ using document root <none> and password file <none>
I20260807 03:09:42.221231 27103 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:42.221385 27103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:42.226575 27103 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260807 03:09:42.229107 27119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:42.230293 27103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:09:42.230522 27103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "427b4da3f6ea4d608e4bbc17cc0629b5"
format_stamp: "Formatted at 2026-08-07 03:09:39 on dist-test-slave-tm1g"
server_key: "df1a22e677fdec3e82999da234f2f9df"
server_key_iv: "7a669ea5a1902b6efe5e3f03e80c3540"
server_key_version: "encryptionkey@0"
I20260807 03:09:42.231420 27103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:42.260845 27103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:42.265134 27103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:42.265411 27103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:42.266361 27103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:42.267622 27103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:42.267745 27103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:42.267879 27103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:42.267956 27103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:42.300442 27103 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:45105
I20260807 03:09:42.300524 27237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:45105 every 8 connection(s)
I20260807 03:09:42.301844 27103 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:09:42.302822 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 27103
Aug 07 03:09:42 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:42 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
W20260807 03:09:42.318954 27241 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:42.304661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:38555 (local address 127.17.31.62:45009)
0807 03:09:42.305208 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:42.305213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:42.306173 (+ 960us) server_negotiation.cc:407] Connection header received
0807 03:09:42.308531 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:42.308536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:42.308605 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:42.308723 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:42.310453 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.311369 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.313027 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.313501 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.313637 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:42.318647 (+ 5010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:38555: BlockingRecv error: recv got EOF from 127.17.31.3:38555 (error 108)
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":293,"threads_started":1}
I20260807 03:09:42.319502 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:42.304036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.17.31.62:45009)
0807 03:09:42.304247 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:42.304257 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:42.304684 (+ 427us) server_negotiation.cc:407] Connection header received
0807 03:09:42.304910 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:42.304918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:42.305056 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:42.305164 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:42.307080 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.308240 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.309456 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.310044 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.310181 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:42.312745 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:42.312756 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:42.312774 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:42.312817 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:42.316162 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:42.316950 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:42.316957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:42.316963 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:42.317133 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:09:42.317528 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:09:42.317532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:09:42.317535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:09:42.317934 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:42.318451 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:42.318772 (+ 321us) server_negotiation.cc:299] Negotiation successful
0807 03:09:42.319114 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 03:09:42.319599 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:42.305252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:38555)
0807 03:09:42.306031 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:42.306074 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:42.308270 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:42.308964 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:42.308972 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:42.309498 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:42.310253 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:42.310277 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.311517 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.311522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:42.312849 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:42.312876 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.313727 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.313737 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:42.313985 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:42.314559 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:42.314579 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:42.318403 (+ 3824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":335,"threads_started":1}
W20260807 03:09:42.320257 27238 heartbeater.cc:644] Failed to heartbeat to 127.17.31.62:45009 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.31.62:45009: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Aug 07 03:09:42 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:42 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:42.328177 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:42.320748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:41459)
0807 03:09:42.320990 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:42.321004 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:42.321072 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:42.321364 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:42.321367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:42.321858 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:42.322133 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:42.322145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.323320 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.323324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:42.324310 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:42.324321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.324954 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.324958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:42.325079 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:42.325538 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:42.325553 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:42.328002 (+ 2449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117}
W20260807 03:09:42.328414 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:42.320868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:41459 (local address 127.17.31.62:45009)
0807 03:09:42.321030 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:42.321035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:42.321052 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:09:42.321146 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:42.321150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:42.321231 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:42.321323 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:42.322261 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.323197 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.324445 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.324842 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.324931 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:42.328191 (+ 3260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:41459: BlockingRecv error: recv got EOF from 127.17.31.3:41459 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 07 03:09:42 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:42 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:42.335816 27242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:42.328750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:51981)
0807 03:09:42.328902 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:42.328918 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:42.328986 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:42.329227 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:42.329231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:42.329487 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:42.329617 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:42.329624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.330555 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.330558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:42.331438 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:42.331448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.332020 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.332023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:42.332139 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:42.332582 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:42.332597 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:42.335595 (+ 2998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 03:09:42.335951 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:42.328834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:51981 (local address 127.17.31.62:45009)
0807 03:09:42.329019 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:42.329024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:42.329042 (+ 18us) server_negotiation.cc:407] Connection header received
0807 03:09:42.329077 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:42.329080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:42.329125 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:42.329199 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:42.329724 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.330446 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.331548 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.331917 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.332000 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:42.335763 (+ 3763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:51981: BlockingRecv error: recv got EOF from 127.17.31.3:51981 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260807 03:09:42.336205 27238 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 07 03:09:42 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:42 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:42.425395 27243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:42.417002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:53791)
0807 03:09:42.417523 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:42.417542 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:42.417682 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:42.418004 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:42.418009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:42.418560 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:42.418771 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:42.418780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.419757 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.419762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:42.420712 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:42.420723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.421519 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.421524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:42.421647 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:42.422281 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:42.422343 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:42.425054 (+ 2711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":257,"threads_started":1}
W20260807 03:09:42.425498 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:42.417084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:53791 (local address 127.17.31.62:45009)
0807 03:09:42.417307 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:42.417315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:42.417635 (+ 320us) server_negotiation.cc:407] Connection header received
0807 03:09:42.417820 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:42.417824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:42.417883 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:42.417974 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:42.418911 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.419619 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.420943 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.421362 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.421470 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:42.425247 (+ 3777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:53791: BlockingRecv error: recv got EOF from 127.17.31.1:53791 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 07 03:09:42 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:42 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:42.881711 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:42.873042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:60353)
0807 03:09:42.873570 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:42.873587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:42.873710 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:42.873990 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:42.873993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:42.874545 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:42.874836 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:42.874847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.876168 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.876172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:42.877298 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:42.877310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.878046 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.878050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:42.878183 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:42.878829 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:42.878847 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:42.881491 (+ 2644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":286,"threads_started":1}
W20260807 03:09:42.881955 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:42.873087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:60353 (local address 127.17.31.62:45009)
0807 03:09:42.873281 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:42.873285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:42.873659 (+ 374us) server_negotiation.cc:407] Connection header received
0807 03:09:42.873801 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:42.873805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:42.873874 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:42.873954 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:42.874977 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.876028 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.877415 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:42.877900 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:42.877994 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:42.881671 (+ 3677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:60353: BlockingRecv error: recv got EOF from 127.17.31.2:60353 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 07 03:09:43 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:43 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:43.345642 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:43.336765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:54165)
0807 03:09:43.337383 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:43.337398 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:43.337506 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:43.337791 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:43.337794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:43.338282 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:43.338605 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:43.338624 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:43.339717 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:43.339721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:43.340830 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:43.340842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:43.341483 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:43.341487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:43.341604 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:43.342210 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:43.342229 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:43.345318 (+ 3089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":368,"threads_started":1}
W20260807 03:09:43.345768 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:43.336872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:54165 (local address 127.17.31.62:45009)
0807 03:09:43.337029 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:43.337033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:43.337465 (+ 432us) server_negotiation.cc:407] Connection header received
0807 03:09:43.337615 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:43.337619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:43.337681 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:43.337755 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:43.338742 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:43.339574 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:43.340947 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:43.341342 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:43.341430 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:43.345525 (+ 4095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:54165: BlockingRecv error: recv got EOF from 127.17.31.3:54165 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 07 03:09:43 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:43 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:43.434223 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:43.426243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:60767)
0807 03:09:43.426801 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:43.426817 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:43.426925 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:43.427197 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:43.427200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:43.427607 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:43.427755 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:43.427763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:43.428740 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:43.428743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:43.429672 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:43.429685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:43.430382 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:43.430386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:43.430500 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:43.431043 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:43.431059 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:43.433815 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":330,"threads_started":1}
W20260807 03:09:43.434322 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:43.426281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:60767 (local address 127.17.31.62:45009)
0807 03:09:43.426471 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:43.426475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:43.426885 (+ 410us) server_negotiation.cc:407] Connection header received
0807 03:09:43.427016 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:43.427019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:43.427103 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:43.427183 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:43.427870 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:43.428619 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:43.429790 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:43.430209 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:43.430335 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:43.434021 (+ 3686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:60767: BlockingRecv error: recv got EOF from 127.17.31.1:60767 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 07 03:09:43 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:43 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:43.895159 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:43.882615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:34915)
0807 03:09:43.883124 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:43.883141 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:43.883302 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:43.883654 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:43.883659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:43.884113 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:43.884263 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:43.884272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:43.885285 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:43.885302 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:43.886219 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:43.886231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:43.886988 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:43.886994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:43.887175 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:43.891874 (+ 4699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:43.891899 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:43.894803 (+ 2904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":274,"threads_started":1}
W20260807 03:09:43.895242 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:43.882662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:34915 (local address 127.17.31.62:45009)
0807 03:09:43.882885 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:43.882894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:43.883233 (+ 339us) server_negotiation.cc:407] Connection header received
0807 03:09:43.883423 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:43.883428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:43.883496 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:43.883604 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:43.884407 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:43.885155 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:43.886424 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:43.886835 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:43.886973 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:43.895024 (+ 8051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:34915: BlockingRecv error: recv got EOF from 127.17.31.2:34915 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 07 03:09:44 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:44 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:44.354662 27249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:44.346673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:53203)
0807 03:09:44.347220 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:44.347242 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:44.347400 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:44.347799 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:44.347805 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:44.348324 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:44.348487 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:44.348497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:44.349467 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:44.349471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:44.350417 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:44.350428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:44.351041 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:44.351044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:44.351153 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:44.351720 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:44.351735 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:44.354424 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":285,"threads_started":1}
W20260807 03:09:44.354816 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:44.346880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:53203 (local address 127.17.31.62:45009)
0807 03:09:44.347087 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:44.347093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:44.347327 (+ 234us) server_negotiation.cc:407] Connection header received
0807 03:09:44.347532 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:44.347543 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:44.347633 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:44.347731 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:44.348649 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:44.349352 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:44.350523 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:44.350916 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:44.351013 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:44.354606 (+ 3593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:53203: BlockingRecv error: recv got EOF from 127.17.31.3:53203 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 07 03:09:44 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:44 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:44.443481 27250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:44.435244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:38693)
0807 03:09:44.435772 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:44.435787 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:44.435954 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:44.436250 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:44.436253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:44.436578 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:44.436743 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:44.436751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:44.437735 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:44.437739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:44.438651 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:44.438661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:44.439420 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:44.439424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:44.439540 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:44.440075 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:44.440091 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:44.443135 (+ 3044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":303,"threads_started":1}
W20260807 03:09:44.443554 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:44.435287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:38693 (local address 127.17.31.62:45009)
0807 03:09:44.435442 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:44.435446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:44.435863 (+ 417us) server_negotiation.cc:407] Connection header received
0807 03:09:44.436072 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:44.436076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:44.436138 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:44.436213 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:44.436859 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:44.437610 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:44.438761 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:44.439298 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:44.439392 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:44.443317 (+ 3925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:38693: BlockingRecv error: recv got EOF from 127.17.31.1:38693 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 07 03:09:44 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:44 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:44.905033 27251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:44.896287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:60719)
0807 03:09:44.896905 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:44.896921 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:44.897080 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:44.897432 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:44.897436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:44.897816 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:44.897982 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:44.897991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:44.899153 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:44.899160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:44.900257 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:44.900269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:44.901003 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:44.901009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:44.901193 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:44.901805 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:44.901823 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:44.904784 (+ 2961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":350,"threads_started":1}
W20260807 03:09:44.905169 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:44.896402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:60719 (local address 127.17.31.62:45009)
0807 03:09:44.896594 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:44.896601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:44.897027 (+ 426us) server_negotiation.cc:407] Connection header received
0807 03:09:44.897195 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:44.897198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:44.897262 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:44.897380 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:44.898132 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:44.899016 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:44.900417 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:44.900848 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:44.900982 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:44.904971 (+ 3989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:60719: BlockingRecv error: recv got EOF from 127.17.31.2:60719 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 07 03:09:45 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:45 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:45.363283 27252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:45.355537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:45463)
0807 03:09:45.356023 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:45.356039 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:45.356158 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:45.356464 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:45.356467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:45.356838 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:45.357024 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:45.357035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:45.358021 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:45.358025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:45.358975 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:45.358987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:45.359624 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:45.359628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:45.359749 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:45.360280 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:45.360297 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:45.362937 (+ 2640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":265,"threads_started":1}
W20260807 03:09:45.363454 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:45.355596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:45463 (local address 127.17.31.62:45009)
0807 03:09:45.355761 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:45.355766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:45.356104 (+ 338us) server_negotiation.cc:407] Connection header received
0807 03:09:45.356253 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:45.356257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:45.356317 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:45.356410 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:45.357190 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:45.357888 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:45.359103 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:45.359512 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:45.359601 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:45.363202 (+ 3601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:45463: BlockingRecv error: recv got EOF from 127.17.31.3:45463 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 07 03:09:45 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:45 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:45.452471 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:45.444593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:55135)
0807 03:09:45.445136 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:45.445152 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:45.445314 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:45.445695 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:45.445700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:45.446207 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:45.446386 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:45.446395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:45.447338 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:45.447342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:45.448282 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:45.448292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:45.448916 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:45.448920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:45.449051 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:45.449596 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:45.449612 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:45.452234 (+ 2622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":310,"threads_started":1}
W20260807 03:09:45.452683 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:45.444672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:55135 (local address 127.17.31.62:45009)
0807 03:09:45.444868 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:45.444874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:45.445249 (+ 375us) server_negotiation.cc:407] Connection header received
0807 03:09:45.445452 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:45.445460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:45.445542 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:45.445648 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:45.446516 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:45.447233 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:45.448392 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:45.448797 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:45.448885 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:45.452404 (+ 3519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:55135: BlockingRecv error: recv got EOF from 127.17.31.1:55135 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 07 03:09:45 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:45 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:45.917985 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:45.906009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:53117)
0807 03:09:45.906493 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:45.906510 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:45.906626 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:45.906917 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:45.906921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:45.907299 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:45.907455 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:45.907464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:45.908416 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:45.908420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:45.909367 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:45.909378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:45.910028 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:45.910032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:45.910163 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:45.913616 (+ 3453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:45.913647 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:45.917610 (+ 3963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":242,"threads_started":1}
W20260807 03:09:45.918140 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:45.906077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:53117 (local address 127.17.31.62:45009)
0807 03:09:45.906319 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:45.906326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:45.906584 (+ 258us) server_negotiation.cc:407] Connection header received
0807 03:09:45.906719 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:45.906722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:45.906785 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:45.906869 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:45.907568 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:45.908304 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:45.909487 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:45.909895 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:45.909995 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:45.917856 (+ 7861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:53117: BlockingRecv error: recv got EOF from 127.17.31.2:53117 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 07 03:09:46 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:46 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:46.372635 27256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:46.364288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:38117)
0807 03:09:46.364864 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:46.364880 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:46.364987 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:46.365338 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:46.365342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:46.365797 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:46.366001 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:46.366012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:46.367045 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:46.367053 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:46.368006 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:46.368025 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:46.368694 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:46.368698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:46.368842 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:46.369510 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:46.369528 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:46.372284 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":321,"threads_started":1}
W20260807 03:09:46.372790 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:46.364406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:38117 (local address 127.17.31.62:45009)
0807 03:09:46.364664 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:46.364671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:46.364941 (+ 270us) server_negotiation.cc:407] Connection header received
0807 03:09:46.365108 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:46.365112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:46.365173 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:46.365251 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:46.366130 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:46.366911 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:46.368139 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:46.368543 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:46.368642 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:46.372533 (+ 3891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:38117: BlockingRecv error: recv got EOF from 127.17.31.3:38117 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 07 03:09:46 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:46 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:46.461724 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:46.453480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:56149)
0807 03:09:46.453965 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:46.453981 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:46.454097 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:46.454380 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:46.454383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:46.454780 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:46.454950 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:46.454958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:46.455900 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:46.455903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:46.456820 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:46.456831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:46.457652 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:46.457656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:46.457849 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:46.458557 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:46.458573 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:46.461481 (+ 2908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":248,"threads_started":1}
W20260807 03:09:46.461866 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:46.453634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:56149 (local address 127.17.31.62:45009)
0807 03:09:46.453857 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:46.453863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:46.454063 (+ 200us) server_negotiation.cc:407] Connection header received
0807 03:09:46.454190 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:46.454193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:46.454252 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:46.454350 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:46.455075 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:46.455788 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:46.456929 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:46.457497 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:46.457622 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:46.461654 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:56149: BlockingRecv error: recv got EOF from 127.17.31.1:56149 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 07 03:09:46 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:46 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:46.927915 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:46.919090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:47545)
0807 03:09:46.919588 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:46.919605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:46.919753 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:46.920177 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:46.920180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:46.920555 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:46.920709 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:46.920718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:46.922092 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:46.922096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:46.923199 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:46.923212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:46.923815 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:46.923818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:46.923930 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:46.924426 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:46.924441 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:46.927565 (+ 3124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":232,"threads_started":1}
W20260807 03:09:46.927989 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:46.919274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:47545 (local address 127.17.31.62:45009)
0807 03:09:46.919476 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:46.919483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:46.919686 (+ 203us) server_negotiation.cc:407] Connection header received
0807 03:09:46.919902 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:46.919910 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:46.920035 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:46.920148 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:46.920876 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:46.921940 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:46.923317 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:46.923699 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:46.923791 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:46.927749 (+ 3958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:47545: BlockingRecv error: recv got EOF from 127.17.31.2:47545 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 07 03:09:47 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:47 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:47.382453 27259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:47.373655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:35527)
0807 03:09:47.374153 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:47.374186 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:47.374365 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:47.374692 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:47.374696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:47.375126 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:47.375295 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:47.375304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:47.376374 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:47.376377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:47.377383 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:47.377394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:47.378023 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:47.378027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:47.378193 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:47.378808 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:47.378832 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:47.382062 (+ 3230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":248,"threads_started":1}
W20260807 03:09:47.382617 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:47.373709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:35527 (local address 127.17.31.62:45009)
0807 03:09:47.373922 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:47.373931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:47.374270 (+ 339us) server_negotiation.cc:407] Connection header received
0807 03:09:47.374492 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:47.374496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:47.374560 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:47.374672 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:47.375437 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:47.376226 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:47.377500 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:47.377907 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:47.378000 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:47.382288 (+ 4288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:35527: BlockingRecv error: recv got EOF from 127.17.31.3:35527 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 07 03:09:47 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:47 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:47.470719 27260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:47.462755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:44403)
0807 03:09:47.463343 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:47.463363 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:47.463524 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:47.463887 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:47.463893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:47.464433 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:47.464640 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:47.464654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:47.465655 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:47.465658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:47.466560 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:47.466571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:47.467225 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:47.467229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:47.467345 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:47.467878 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:47.467894 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:47.470473 (+ 2579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":310,"threads_started":1}
W20260807 03:09:47.470989 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:47.462835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:44403 (local address 127.17.31.62:45009)
0807 03:09:47.463037 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:47.463045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:47.463446 (+ 401us) server_negotiation.cc:407] Connection header received
0807 03:09:47.463653 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:47.463657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:47.463717 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:47.463817 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:47.464775 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:47.465543 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:47.466672 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:47.467094 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:47.467186 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:47.470717 (+ 3531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:44403: BlockingRecv error: recv got EOF from 127.17.31.1:44403 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 07 03:09:47 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:47 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:47.937104 27261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:47.928823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:48317)
0807 03:09:47.929438 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:47.929455 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:47.929568 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:47.929870 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:47.929873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:47.930252 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:47.930609 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:47.930620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:47.931586 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:47.931590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:47.932541 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:47.932556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:47.933507 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:47.933516 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:47.933655 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:47.934274 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:47.934291 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:47.936868 (+ 2577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":329,"threads_started":1}
W20260807 03:09:47.937284 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:47.928901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:48317 (local address 127.17.31.62:45009)
0807 03:09:47.929062 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:47.929067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:47.929523 (+ 456us) server_negotiation.cc:407] Connection header received
0807 03:09:47.929657 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:47.929660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:47.929738 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:47.929825 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:47.930737 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:47.931473 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:47.932694 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:47.933332 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:47.933461 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:47.937038 (+ 3577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:48317: BlockingRecv error: recv got EOF from 127.17.31.2:48317 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 07 03:09:48 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:48 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:48.391644 27262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:48.383361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:50477)
0807 03:09:48.383856 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:48.383873 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:48.383986 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:48.384294 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:48.384297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:48.384688 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:48.384851 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:48.384860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:48.385876 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:48.385879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:48.386865 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:48.386876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:48.387573 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:48.387577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:48.387703 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:48.388315 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:48.388333 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:48.391369 (+ 3036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":293,"threads_started":1}
W20260807 03:09:48.391839 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:48.383438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:50477 (local address 127.17.31.62:45009)
0807 03:09:48.383679 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:48.383687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:48.383948 (+ 261us) server_negotiation.cc:407] Connection header received
0807 03:09:48.384081 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:48.384085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:48.384148 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:48.384230 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:48.384975 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:48.385734 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:48.386985 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:48.387420 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:48.387513 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:48.391590 (+ 4077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:50477: BlockingRecv error: recv got EOF from 127.17.31.3:50477 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 07 03:09:48 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:48 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:48.480115 27263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:48.471595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:41725)
0807 03:09:48.472226 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:48.472242 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:48.472393 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:48.472755 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:48.472763 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:48.473197 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:48.473368 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:48.473377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:48.474446 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:48.474451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:48.475388 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:48.475400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:48.476214 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:48.476220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:48.476414 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:48.477071 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:48.477089 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:48.479807 (+ 2718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":332,"threads_started":1}
W20260807 03:09:48.480170 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:48.471648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:41725 (local address 127.17.31.62:45009)
0807 03:09:48.471833 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:48.471839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:48.472333 (+ 494us) server_negotiation.cc:407] Connection header received
0807 03:09:48.472525 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:48.472531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:48.472602 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:48.472699 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:48.473532 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:48.474276 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:48.475538 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:48.476070 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:48.476200 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:48.479995 (+ 3795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:41725: BlockingRecv error: recv got EOF from 127.17.31.1:41725 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 07 03:09:48 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:48 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:48.952000 27264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:48.937989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:52401)
0807 03:09:48.938665 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:48.938688 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:48.938814 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:48.939177 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:48.939182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:48.939776 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:48.940018 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:48.940031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:48.941115 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:48.941119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:48.942024 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:48.942034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:48.942771 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:48.942776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:48.942886 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:48.948657 (+ 5771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:48.948674 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:48.951654 (+ 2980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":327,"threads_started":1}
W20260807 03:09:48.952261 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:48.938030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:52401 (local address 127.17.31.62:45009)
0807 03:09:48.938219 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:48.938224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:48.938774 (+ 550us) server_negotiation.cc:407] Connection header received
0807 03:09:48.938937 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:48.938941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:48.939004 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:48.939092 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:48.940186 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:48.940992 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:48.942164 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:48.942651 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:48.942750 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:48.951925 (+ 9175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:52401: BlockingRecv error: recv got EOF from 127.17.31.2:52401 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 07 03:09:49 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:49 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:49.400951 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:49.392591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:55509)
0807 03:09:49.393227 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:49.393243 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:49.393354 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:49.393717 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:49.393721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:49.394171 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:49.394460 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:49.394478 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:49.395532 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:49.395538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:49.396451 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:49.396462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:49.397180 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:49.397186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:49.397352 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:49.397887 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:49.397905 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:49.400613 (+ 2708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":284,"threads_started":1}
W20260807 03:09:49.401057 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:49.392650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:55509 (local address 127.17.31.62:45009)
0807 03:09:49.392876 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:49.392883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:49.393314 (+ 431us) server_negotiation.cc:407] Connection header received
0807 03:09:49.393470 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:49.393474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:49.393536 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:49.393628 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:49.394620 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:49.395397 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:49.396588 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:49.397034 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:49.397161 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:49.400811 (+ 3650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:55509: BlockingRecv error: recv got EOF from 127.17.31.3:55509 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 07 03:09:49 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:49 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:49.489408 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:49.480979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:58009)
0807 03:09:49.481531 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:49.481546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:49.481692 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:49.482029 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:49.482035 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:49.482509 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:49.482663 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:49.482673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:49.483703 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:49.483709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:49.484621 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:49.484632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:49.485374 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:49.485380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:49.485560 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:49.486188 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:49.486205 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:49.489064 (+ 2859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":279,"threads_started":1}
W20260807 03:09:49.489488 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:49.481200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:58009 (local address 127.17.31.62:45009)
0807 03:09:49.481409 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:49.481418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:49.481626 (+ 208us) server_negotiation.cc:407] Connection header received
0807 03:09:49.481817 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:49.481827 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:49.481887 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:49.481994 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:49.482806 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:49.483560 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:49.484755 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:49.485179 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:49.485314 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:49.489267 (+ 3953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:58009: BlockingRecv error: recv got EOF from 127.17.31.1:58009 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 07 03:09:49 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:49 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:49.962119 27267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:49.952972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:49361)
0807 03:09:49.953480 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:49.953495 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:49.953646 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:49.953977 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:49.953982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:49.954449 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:49.954613 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:49.954621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:49.956343 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:49.956348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:49.957607 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:49.957620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:49.958417 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:49.958423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:49.958603 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:49.959156 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:49.959173 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:49.961884 (+ 2711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":265,"threads_started":1}
W20260807 03:09:49.962273 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:49.953062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:49361 (local address 127.17.31.62:45009)
0807 03:09:49.953293 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:49.953301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:49.953581 (+ 280us) server_negotiation.cc:407] Connection header received
0807 03:09:49.953767 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:49.953773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:49.953840 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:49.953938 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:49.954752 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:49.956163 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:49.957746 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:49.958211 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:49.958361 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:49.962076 (+ 3715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:49361: BlockingRecv error: recv got EOF from 127.17.31.2:49361 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 07 03:09:50 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:50 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:50.410290 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:50.402111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:36973)
0807 03:09:50.402705 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:50.402725 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:50.402892 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:50.403323 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:50.403328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:50.403742 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:50.403900 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:50.403908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:50.404839 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:50.404842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:50.405753 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:50.405763 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:50.406483 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:50.406487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:50.406605 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:50.407181 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:50.407198 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:50.409937 (+ 2739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":311,"threads_started":1}
W20260807 03:09:50.410426 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:50.402210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:36973 (local address 127.17.31.62:45009)
0807 03:09:50.402477 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:50.402483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:50.402831 (+ 348us) server_negotiation.cc:407] Connection header received
0807 03:09:50.403011 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:50.403032 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:50.403132 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:50.403244 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:50.404026 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:50.404714 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:50.405871 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:50.406289 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:50.406443 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:50.410145 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:36973: BlockingRecv error: recv got EOF from 127.17.31.3:36973 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 07 03:09:50 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:50 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:50.498271 27269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:50.490244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:54131)
0807 03:09:50.490739 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:50.490755 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:50.490879 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:50.491150 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:50.491153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:50.491541 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:50.491705 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:50.491714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:50.492785 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:50.492793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:50.493757 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:50.493770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:50.494461 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:50.494466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:50.494585 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:50.495127 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:50.495144 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:50.498021 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":278,"threads_started":1}
W20260807 03:09:50.498505 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:50.490388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:54131 (local address 127.17.31.62:45009)
0807 03:09:50.490641 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:50.490647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:50.490831 (+ 184us) server_negotiation.cc:407] Connection header received
0807 03:09:50.490964 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:50.490968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:50.491029 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:50.491105 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:50.491824 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:50.492619 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:50.493883 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:50.494327 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:50.494415 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:50.498205 (+ 3790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:54131: BlockingRecv error: recv got EOF from 127.17.31.1:54131 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 07 03:09:50 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:50 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:50.971882 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:50.963137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:38305)
0807 03:09:50.963698 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:50.963714 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:50.963824 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:50.964113 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:50.964116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:50.964539 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:50.964779 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:50.964794 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:50.965847 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:50.965852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:50.967007 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:50.967022 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:50.967959 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:50.967967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:50.968155 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:50.968715 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:50.968732 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:50.971639 (+ 2907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":320,"threads_started":1}
W20260807 03:09:50.972044 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:50.963190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:38305 (local address 127.17.31.62:45009)
0807 03:09:50.963383 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:50.963389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:50.963781 (+ 392us) server_negotiation.cc:407] Connection header received
0807 03:09:50.963914 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:50.963918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:50.963980 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:50.964061 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:50.964916 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:50.965690 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:50.967180 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:50.967780 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:50.967922 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:50.971827 (+ 3905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:38305: BlockingRecv error: recv got EOF from 127.17.31.2:38305 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 07 03:09:51 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:51 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:51.420071 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:51.411224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:45669)
0807 03:09:51.411780 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:51.411797 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:51.411967 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:51.412414 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:51.412417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:51.413007 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:51.413270 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:51.413281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:51.414472 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:51.414476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:51.415406 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:51.415417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:51.416106 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:51.416110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:51.416239 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:51.416831 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:51.416849 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:51.419824 (+ 2975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":283,"threads_started":1}
W20260807 03:09:51.420253 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:51.411293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:45669 (local address 127.17.31.62:45009)
0807 03:09:51.411581 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:51.411588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:51.411896 (+ 308us) server_negotiation.cc:407] Connection header received
0807 03:09:51.412093 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:51.412099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:51.412215 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:51.412330 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:51.413430 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:51.414325 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:51.415539 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:51.415953 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:51.416047 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:51.420016 (+ 3969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:45669: BlockingRecv error: recv got EOF from 127.17.31.3:45669 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 07 03:09:51 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:51 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:51.507725 27272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:51.499231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:40473)
0807 03:09:51.499754 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:51.499771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:51.499878 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:51.500165 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:51.500169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:51.500542 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:51.500690 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:51.500698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:51.501711 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:51.501715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:51.502669 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:51.502681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:51.503590 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:51.503599 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:51.503838 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:51.504529 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:51.504546 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:51.507453 (+ 2907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":298,"threads_started":1}
W20260807 03:09:51.508136 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:51.499298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:40473 (local address 127.17.31.62:45009)
0807 03:09:51.499475 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:51.499479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:51.499845 (+ 366us) server_negotiation.cc:407] Connection header received
0807 03:09:51.499971 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:51.499974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:51.500042 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:51.500124 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:51.500816 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:51.501593 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:51.502787 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:51.503428 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:51.503552 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:51.507848 (+ 4296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:40473: BlockingRecv error: recv got EOF from 127.17.31.1:40473 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 03:09:51 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:51 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:51.981066 27273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:51.972878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:33139)
0807 03:09:51.973367 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:51.973383 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:51.973540 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:51.973870 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:51.973874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:51.974288 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:51.974510 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:51.974520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:51.975558 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:51.975576 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:51.976564 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:51.976599 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:51.977319 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:51.977325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:51.977511 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:51.978127 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:51.978144 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:51.980843 (+ 2699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":294,"threads_started":1}
W20260807 03:09:51.981256 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:51.972925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:33139 (local address 127.17.31.62:45009)
0807 03:09:51.973138 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:51.973146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:51.973464 (+ 318us) server_negotiation.cc:407] Connection header received
0807 03:09:51.973657 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:51.973669 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:51.973730 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:51.973845 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:51.974653 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:51.975424 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:51.976735 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:51.977162 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:51.977308 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:51.981032 (+ 3724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:33139: BlockingRecv error: recv got EOF from 127.17.31.2:33139 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 07 03:09:52 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:52 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:52.430198 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:52.421120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:35935)
0807 03:09:52.421684 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:52.421701 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:52.421812 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:52.422140 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:52.422144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:52.422590 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:52.422751 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:52.422759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:52.423821 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:52.423831 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:52.424767 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:52.424778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:52.425872 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:52.425879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:52.426077 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:52.426894 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:52.426919 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:52.429831 (+ 2912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":336,"threads_started":1}
W20260807 03:09:52.430269 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:52.421290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:35935 (local address 127.17.31.62:45009)
0807 03:09:52.421529 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:52.421534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:52.421780 (+ 246us) server_negotiation.cc:407] Connection header received
0807 03:09:52.421918 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:52.421922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:52.421983 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:52.422101 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:52.422907 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:52.423665 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:52.424937 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:52.425554 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:52.425721 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:52.430036 (+ 4315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:35935: BlockingRecv error: recv got EOF from 127.17.31.3:35935 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 07 03:09:52 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:52 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:52.517328 27275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:52.508742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:51859)
0807 03:09:52.509298 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:52.509317 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:52.509472 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:52.509859 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:52.509865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:52.510331 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:52.510508 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:52.510517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:52.511564 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:52.511572 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:52.512477 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:52.512487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:52.513419 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:52.513425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:52.513601 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:52.514163 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:52.514181 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:52.516971 (+ 2790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":296,"threads_started":1}
W20260807 03:09:52.517455 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:52.508851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:51859 (local address 127.17.31.62:45009)
0807 03:09:52.509050 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:52.509059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:52.509409 (+ 350us) server_negotiation.cc:407] Connection header received
0807 03:09:52.509607 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:52.509612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:52.509700 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:52.509809 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:52.510654 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:52.511423 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:52.512619 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:52.513252 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:52.513388 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:52.517194 (+ 3806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:51859: BlockingRecv error: recv got EOF from 127.17.31.1:51859 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 07 03:09:52 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:52 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:52.990347 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:52.982059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:34193)
0807 03:09:52.982623 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:52.982639 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:52.982752 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:52.983042 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:52.983045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:52.983443 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:52.983633 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:52.983641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:52.984591 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:52.984594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:52.985489 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:52.985499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:52.986138 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:52.986142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:52.986254 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:52.986840 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:52.986857 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:52.989972 (+ 3115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":333,"threads_started":1}
W20260807 03:09:52.990423 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:52.982096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:34193 (local address 127.17.31.62:45009)
0807 03:09:52.982262 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:52.982267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:52.982713 (+ 446us) server_negotiation.cc:407] Connection header received
0807 03:09:52.982846 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:52.982850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:52.982912 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:52.983002 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:52.983751 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:52.984476 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:52.985603 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:52.986015 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:52.986102 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:52.990149 (+ 4047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:34193: BlockingRecv error: recv got EOF from 127.17.31.2:34193 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 07 03:09:53 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:53 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:53.439622 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:53.431112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:36615)
0807 03:09:53.431670 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:53.431686 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:53.431797 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:53.432075 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:53.432078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:53.432477 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:53.432731 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:53.432744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:53.433971 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:53.433976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:53.435070 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:53.435083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:53.435755 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:53.435776 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:53.435912 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:53.436445 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:53.436462 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:53.439251 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":318,"threads_started":1}
W20260807 03:09:53.439716 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:53.431190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:36615 (local address 127.17.31.62:45009)
0807 03:09:53.431381 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:53.431386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:53.431756 (+ 370us) server_negotiation.cc:407] Connection header received
0807 03:09:53.431891 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:53.431895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:53.431955 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:53.432034 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:53.432882 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:53.433814 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:53.435189 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:53.435593 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:53.435734 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:53.439466 (+ 3732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:36615: BlockingRecv error: recv got EOF from 127.17.31.3:36615 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 07 03:09:53 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:53 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:53.526497 27278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:53.518397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:57039)
0807 03:09:53.518885 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:53.518901 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:53.519023 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:53.519307 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:53.519310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:53.519683 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:53.519839 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:53.519847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:53.520821 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:53.520824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:53.521691 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:53.521701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:53.522621 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:53.522625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:53.522752 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:53.523345 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:53.523362 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:53.526129 (+ 2767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":272,"threads_started":1}
W20260807 03:09:53.526768 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:53.518455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:57039 (local address 127.17.31.62:45009)
0807 03:09:53.518672 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:53.518682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:53.518977 (+ 295us) server_negotiation.cc:407] Connection header received
0807 03:09:53.519116 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:53.519120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:53.519179 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:53.519257 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:53.519957 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:53.520696 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:53.521870 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:53.522498 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:53.522590 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:53.526402 (+ 3812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:57039: BlockingRecv error: recv got EOF from 127.17.31.1:57039 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 07 03:09:53 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:53 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:54.004184 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:53.991304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:50201)
0807 03:09:53.991812 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:53.991828 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:53.991980 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:53.992319 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:53.992324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:53.992720 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:53.992880 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:53.992888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:53.993945 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:53.993950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:53.995108 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:53.995120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:53.995835 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:53.995840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:53.996026 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:54.001187 (+ 5161us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:54.001206 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:54.003949 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":273,"threads_started":1}
W20260807 03:09:54.004443 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:53.991351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:50201 (local address 127.17.31.62:45009)
0807 03:09:53.991581 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:53.991588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:53.991910 (+ 322us) server_negotiation.cc:407] Connection header received
0807 03:09:53.992110 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:53.992115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:53.992173 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:53.992273 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:53.993015 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:53.993795 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:53.995236 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:53.995685 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:53.995824 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:54.004170 (+ 8346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:50201: BlockingRecv error: recv got EOF from 127.17.31.2:50201 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 07 03:09:54 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:54 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:54.448447 27282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:54.440591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:56241)
0807 03:09:54.441138 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:54.441153 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:54.441274 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:54.441554 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:54.441558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:54.441926 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:54.442089 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:54.442107 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:54.443096 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:54.443100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:54.444053 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:54.444064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:54.444714 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:54.444718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:54.444846 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:54.445499 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:54.445517 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:54.448204 (+ 2687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":295,"threads_started":1}
W20260807 03:09:54.448662 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:54.440644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:56241 (local address 127.17.31.62:45009)
0807 03:09:54.440829 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:54.440834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:54.441219 (+ 385us) server_negotiation.cc:407] Connection header received
0807 03:09:54.441372 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:54.441376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:54.441435 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:54.441516 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:54.442215 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:54.442975 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:54.444188 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:54.444601 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:54.444689 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:54.448381 (+ 3692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:56241: BlockingRecv error: recv got EOF from 127.17.31.3:56241 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 07 03:09:54 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:54 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:54.535861 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:54.527333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:48217)
0807 03:09:54.527893 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:54.527909 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:54.528021 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:54.528293 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:54.528296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:54.528660 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:54.528816 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:54.528825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:54.529764 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:54.529767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:54.530767 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:54.530779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:54.531729 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:54.531735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:54.531916 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:54.532614 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:54.532631 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:54.535589 (+ 2958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":297,"threads_started":1}
W20260807 03:09:54.536029 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:54.527433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:48217 (local address 127.17.31.62:45009)
0807 03:09:54.527610 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:54.527615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:54.527979 (+ 364us) server_negotiation.cc:407] Connection header received
0807 03:09:54.528111 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:54.528115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:54.528174 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:54.528253 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:54.528941 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:54.529650 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:54.530921 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:54.531534 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:54.531665 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:54.535783 (+ 4118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:48217: BlockingRecv error: recv got EOF from 127.17.31.1:48217 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 07 03:09:55 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:55 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:55.013336 27284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:55.005139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:48973)
0807 03:09:55.005719 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:55.005736 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:55.005846 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:55.006111 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:55.006114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:55.006501 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:55.006660 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:55.006668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:55.007653 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:55.007658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:55.008577 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:55.008588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:55.009265 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:55.009269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:55.009389 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:55.010075 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:55.010092 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:55.013074 (+ 2982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":315,"threads_started":1}
W20260807 03:09:55.013496 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:55.005282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:48973 (local address 127.17.31.62:45009)
0807 03:09:55.005509 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:55.005513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:55.005847 (+ 334us) server_negotiation.cc:407] Connection header received
0807 03:09:55.005941 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:55.005945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:55.006023 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:55.006099 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:55.006785 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:55.007537 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:55.008698 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:55.009118 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:55.009210 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:55.013261 (+ 4051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:48973: BlockingRecv error: recv got EOF from 127.17.31.2:48973 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Aug 07 03:09:55 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:55 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:55.457595 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:55.449373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:50883)
0807 03:09:55.449925 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:55.449941 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:55.450053 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:55.450390 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:55.450394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:55.450771 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:55.450933 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:55.450940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:55.451944 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:55.451948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:55.452883 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:55.452893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:55.453581 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:55.453586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:55.453722 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:55.454276 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:55.454293 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:55.457219 (+ 2926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":287,"threads_started":1}
W20260807 03:09:55.457762 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:55.449443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:50883 (local address 127.17.31.62:45009)
0807 03:09:55.449627 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:55.449632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:55.450013 (+ 381us) server_negotiation.cc:407] Connection header received
0807 03:09:55.450149 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:55.450153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:55.450212 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:55.450290 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:55.451054 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:55.451827 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:55.453003 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:55.453405 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:55.453504 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:55.457424 (+ 3920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:50883: BlockingRecv error: recv got EOF from 127.17.31.3:50883 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 07 03:09:55 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:55 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:55.545089 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:55.536687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:54847)
0807 03:09:55.537193 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:55.537208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:55.537367 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:55.537713 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:55.537717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:55.538196 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:55.538465 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:55.538479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:55.539948 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:55.539952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:55.540858 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:55.540869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:55.541502 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:55.541506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:55.541624 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:55.542201 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:55.542218 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:55.544828 (+ 2610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":263,"threads_started":1}
W20260807 03:09:55.545229 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:55.536745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:54847 (local address 127.17.31.62:45009)
0807 03:09:55.537056 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:55.537062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:55.537296 (+ 234us) server_negotiation.cc:407] Connection header received
0807 03:09:55.537502 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:55.537507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:55.537577 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:55.537681 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:55.538651 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:55.539819 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:55.540978 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:55.541374 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:55.541478 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:55.545018 (+ 3540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:54847: BlockingRecv error: recv got EOF from 127.17.31.1:54847 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Aug 07 03:09:56 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:56 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:56.023052 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:56.014288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:42211)
0807 03:09:56.014820 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:56.014835 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:56.014989 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:56.015315 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:56.015319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:56.015710 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:56.015879 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:56.015888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:56.017235 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:56.017242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:56.018521 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:56.018541 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:56.019191 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:56.019195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:56.019310 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:56.019880 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:56.019897 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:56.022754 (+ 2857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
W20260807 03:09:56.023680 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:56.014413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:42211 (local address 127.17.31.62:45009)
0807 03:09:56.014623 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:56.014631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:56.014918 (+ 287us) server_negotiation.cc:407] Connection header received
0807 03:09:56.015123 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:56.015127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:56.015187 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:56.015285 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:56.016048 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:56.017056 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:56.018648 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:56.019071 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:56.019166 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:56.023502 (+ 4336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:42211: BlockingRecv error: recv got EOF from 127.17.31.2:42211 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 07 03:09:56 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:56 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.3@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:56.467123 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:56.458652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.3:60001)
0807 03:09:56.459135 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:56.459150 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:56.459302 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:56.459617 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:56.459621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:56.460150 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:56.460401 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:56.460413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:56.461559 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:56.461564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:56.462581 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:56.462593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:56.463314 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:56.463319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:56.463502 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:56.464065 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:56.464082 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:56.466792 (+ 2710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":243,"threads_started":1}
W20260807 03:09:56.467263 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:56.458745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:60001 (local address 127.17.31.62:45009)
0807 03:09:56.459001 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:56.459008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:56.459229 (+ 221us) server_negotiation.cc:407] Connection header received
0807 03:09:56.459418 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:56.459430 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:56.459491 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:56.459589 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:56.460529 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:56.461416 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:56.462712 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:56.463164 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:56.463292 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:56.467016 (+ 3724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.3:60001: BlockingRecv error: recv got EOF from 127.17.31.3:60001 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 07 03:09:56 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:56 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.1@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:56.554872 27289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:56.546282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.1:37613)
0807 03:09:56.546831 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:56.546847 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:56.547000 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:56.547347 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:56.547352 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:56.547809 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:56.548003 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:56.548012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:56.549060 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:56.549066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:56.549997 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:56.550008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:56.550770 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:56.550776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:56.550954 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:56.551549 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:56.551566 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:56.554469 (+ 2903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":294,"threads_started":1}
W20260807 03:09:56.554994 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:56.546379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:37613 (local address 127.17.31.62:45009)
0807 03:09:56.546652 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:56.546661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:56.546935 (+ 274us) server_negotiation.cc:407] Connection header received
0807 03:09:56.547121 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:56.547126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:56.547195 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:56.547326 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:56.548149 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:56.548887 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:56.550136 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:56.550618 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:56.550746 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:56.554721 (+ 3975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.1:37613: BlockingRecv error: recv got EOF from 127.17.31.1:37613 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 07 03:09:57 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
Aug 07 03:09:57 dist-test-slave-tm1g krb5kdc[26187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.31.2@KRBTEST.COM for oryx/127.17.31.62@KRBTEST.COM, Server not found in Kerberos database
I20260807 03:09:57.032590 27290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:57.024307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:45009 (local address 127.17.31.2:33283)
0807 03:09:57.025086 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:57.025101 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:57.025218 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:57.025514 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:57.025518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:57.025926 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:57.026135 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:57.026144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:57.027175 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:57.027178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:57.028095 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:57.028106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:57.028756 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:57.028760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:57.028934 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:57.029610 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:09:57.029639 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:57.032338 (+ 2699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.31.62:45009: Server oryx/127.17.31.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":541,"threads_started":1}
W20260807 03:09:57.032769 26963 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:57.024417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:33283 (local address 127.17.31.62:45009)
0807 03:09:57.024619 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:57.024623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:57.025176 (+ 553us) server_negotiation.cc:407] Connection header received
0807 03:09:57.025324 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:57.025327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:57.025390 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:57.025473 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:57.026250 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:57.027049 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:57.028208 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:57.028643 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:57.028732 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:57.032531 (+ 3799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.31.2:33283: BlockingRecv error: recv got EOF from 127.17.31.2:33283 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 03:09:57.304873 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 26808
I20260807 03:09:57.316871 27292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:57.305873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.17.31.1:38549)
0807 03:09:57.306494 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:57.306502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:57.306531 (+ 29us) server_negotiation.cc:407] Connection header received
0807 03:09:57.306595 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:57.306602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:57.306834 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:57.307008 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:57.308424 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:57.309462 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:57.310708 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:57.311265 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:57.311371 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:57.313805 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:57.313817 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:57.313837 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:57.313883 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:57.316380 (+ 2497us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.31.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 03:09:57.316639 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35182: Request ticket server kudu/127.17.31.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":306,"threads_started":1}
W20260807 03:09:57.317143 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 26808: Not authorized: Client connection negotiation failed: client connection to 127.17.31.1:38549: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.31.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 03:09:57.317374 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 26808
W20260807 03:09:57.316995 27292 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35182: Request ticket server kudu/127.17.31.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 03:09:57.336061 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 26965
I20260807 03:09:57.348124 27293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:57.337267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.17.31.2:41333)
0807 03:09:57.337940 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:57.337948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:57.337980 (+ 32us) server_negotiation.cc:407] Connection header received
0807 03:09:57.338040 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:57.338047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:57.338381 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:57.338549 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:57.339911 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:57.340996 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:57.342178 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:57.342806 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:57.342915 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:57.345212 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:57.345220 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:57.345232 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:57.345271 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:57.347583 (+ 2312us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.31.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 03:09:57.347831 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51440: Request ticket server kudu/127.17.31.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":346,"threads_started":1}
W20260807 03:09:57.348234 27293 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51440: Request ticket server kudu/127.17.31.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 03:09:57.348219 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 26965: Not authorized: Client connection negotiation failed: client connection to 127.17.31.2:41333: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.31.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 03:09:57.348400 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 26965
I20260807 03:09:57.366549 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 27103
I20260807 03:09:57.378188 27294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:57.367473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.17.31.3:45105)
0807 03:09:57.367966 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:57.367974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:57.367999 (+ 25us) server_negotiation.cc:407] Connection header received
0807 03:09:57.368063 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:57.368069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:57.368296 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:57.368450 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:57.369676 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:57.370838 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:57.371996 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:57.372546 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:57.372641 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:57.375192 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:57.375200 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:57.375213 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:09:57.375247 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:57.377676 (+ 2429us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.31.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0807 03:09:57.377912 (+ 236us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60258: Request ticket server kudu/127.17.31.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":246,"threads_started":1}
W20260807 03:09:57.378295 27294 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60258: Request ticket server kudu/127.17.31.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260807 03:09:57.378435 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 27103: Not authorized: Client connection negotiation failed: client connection to 127.17.31.3:45105: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.31.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260807 03:09:57.378583 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 27103
I20260807 03:09:57.395586 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 26736
I20260807 03:09:57.396857 26800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:57.516726 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 26736
2026-08-07T03:09:57Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21084 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-07T03:09:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:09:57Z Disabled control of system clock
I20260807 03:09:57.598934 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:37125
--webserver_interface=127.17.31.62
--webserver_port=0
--builtin_ntp_servers=127.17.31.20:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:37125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 03:09:57.938199 27299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:57.938503 27299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:57.938585 27299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:57.945703 27299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:09:57.945809 27299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:57.945848 27299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:57.945914 27299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:09:57.945960 27299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:09:57.954871 27299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:37125
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:37125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:57.956146 27299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:57.957883 27299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:57.969527 27307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:57.969588 27304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:57.969493 27305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:57.970512 27299 server_base.cc:1034] running on GCE node
I20260807 03:09:57.971310 27299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:57.972653 27299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:57.973847 27299 hybrid_clock.cc:648] HybridClock initialized: now 1786072197973798 us; error 39 us; skew 500 ppm
I20260807 03:09:57.974201 27299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:09:57.976435 27299 webserver.cc:460] Webserver started at http://127.17.31.62:37235/ using document root <none> and password file <none>
I20260807 03:09:57.977137 27299 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:57.977254 27299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:57.977607 27299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:57.980661 27299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "07142170010645beb1f7c1989804c00b"
format_stamp: "Formatted at 2026-08-07 03:09:57 on dist-test-slave-tm1g"
server_key: "0b6320f505d6612df75a0cd0a900b64a"
server_key_iv: "cae97e3e4e15a5bfc3e9773a07f5c66b"
server_key_version: "encryptionkey@0"
I20260807 03:09:57.981454 27299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "07142170010645beb1f7c1989804c00b"
format_stamp: "Formatted at 2026-08-07 03:09:57 on dist-test-slave-tm1g"
server_key: "0b6320f505d6612df75a0cd0a900b64a"
server_key_iv: "cae97e3e4e15a5bfc3e9773a07f5c66b"
server_key_version: "encryptionkey@0"
I20260807 03:09:57.986370 27299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260807 03:09:57.989658 27313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:57.990964 27299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 03:09:57.991149 27299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "07142170010645beb1f7c1989804c00b"
format_stamp: "Formatted at 2026-08-07 03:09:57 on dist-test-slave-tm1g"
server_key: "0b6320f505d6612df75a0cd0a900b64a"
server_key_iv: "cae97e3e4e15a5bfc3e9773a07f5c66b"
server_key_version: "encryptionkey@0"
I20260807 03:09:57.991364 27299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:58.003270 27299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:58.004024 27299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:58.004312 27299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:58.023125 27299 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:37125
I20260807 03:09:58.023144 27366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:37125 every 8 connection(s)
I20260807 03:09:58.024461 27299 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:09:58.028179 27367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:09:58.029070 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 27299
I20260807 03:09:58.029296 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:09:58.029965 17532 external_mini_cluster.cc:1428] Setting key 21490adf2ffc4b07dd7026fa832a9c60
I20260807 03:09:58.041942 27367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b: Bootstrap starting.
I20260807 03:09:58.043853 27370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:58.032063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.17.31.62:37125)
0807 03:09:58.033088 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:58.033099 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:58.033143 (+ 44us) server_negotiation.cc:407] Connection header received
0807 03:09:58.034767 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:58.034792 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:58.035243 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:58.035554 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:58.036207 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:58.037399 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:58.039438 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:58.040131 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:58.040281 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:58.041537 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:58.041543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:58.041556 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:58.042127 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:58.042225 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:58.042245 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:58.042405 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:58.042591 (+ 186us) server_negotiation.cc:299] Negotiation successful
0807 03:09:58.042928 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":372,"threads_started":1}
I20260807 03:09:58.045516 27367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b: Neither blocks nor log segments found. Creating new log.
I20260807 03:09:58.046685 27367 log.cc:824] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b: Log is configured to *not* fsync() on all Append() calls
I20260807 03:09:58.049263 27367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b: No bootstrap required, opened a new log
I20260807 03:09:58.056373 27367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07142170010645beb1f7c1989804c00b" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37125 } }
I20260807 03:09:58.056809 27367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:09:58.056958 27367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07142170010645beb1f7c1989804c00b, State: Initialized, Role: FOLLOWER
I20260807 03:09:58.057567 27367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07142170010645beb1f7c1989804c00b" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37125 } }
I20260807 03:09:58.057807 27367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:09:58.057950 27367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:09:58.058120 27367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:09:58.060222 27367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07142170010645beb1f7c1989804c00b" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37125 } }
I20260807 03:09:58.060698 27367 leader_election.cc:302] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07142170010645beb1f7c1989804c00b; no voters:
I20260807 03:09:58.061246 27367 leader_election.cc:288] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:09:58.061383 27372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:09:58.062107 27372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [term 1 LEADER]: Becoming Leader. State: Replica: 07142170010645beb1f7c1989804c00b, State: Running, Role: LEADER
I20260807 03:09:58.062716 27372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07142170010645beb1f7c1989804c00b" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37125 } }
I20260807 03:09:58.063330 27367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:09:58.066239 27374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07142170010645beb1f7c1989804c00b. Latest consensus state: current_term: 1 leader_uuid: "07142170010645beb1f7c1989804c00b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07142170010645beb1f7c1989804c00b" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37125 } } }
I20260807 03:09:58.066744 27374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:58.067714 27380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:09:58.072854 27380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:09:58.067538 27373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07142170010645beb1f7c1989804c00b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07142170010645beb1f7c1989804c00b" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37125 } } }
I20260807 03:09:58.074314 27373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b [sys.catalog]: This master's current role is: LEADER
I20260807 03:09:58.085960 27380 catalog_manager.cc:1347] Generated new cluster ID: 8c4b7a1c572e44d19cc7d742ce5b6deb
I20260807 03:09:58.086081 27380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:09:58.125200 27380 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:09:58.126745 27380 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:09:58.136407 27380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07142170010645beb1f7c1989804c00b: Generated new TSK 0
I20260807 03:09:58.137483 27380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:09:58.144017 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37125
--builtin_ntp_servers=127.17.31.20:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260807 03:09:58.484683 27391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:58.484959 27391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:58.485101 27391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:58.492378 27391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:58.492476 27391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:58.492635 27391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:09:58.501624 27391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:58.502940 27391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:58.504735 27391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:58.517948 27397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:58.517994 27396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:58.518030 27399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:58.519382 27391 server_base.cc:1034] running on GCE node
I20260807 03:09:58.519982 27391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:58.520946 27391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:58.522210 27391 hybrid_clock.cc:648] HybridClock initialized: now 1786072198522163 us; error 58 us; skew 500 ppm
I20260807 03:09:58.522562 27391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:09:58.524751 27391 webserver.cc:460] Webserver started at http://127.17.31.1:33313/ using document root <none> and password file <none>
I20260807 03:09:58.525476 27391 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:58.525607 27391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:58.525943 27391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:58.528774 27391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "49b888fb6b624176a09ff790368af10d"
format_stamp: "Formatted at 2026-08-07 03:09:58 on dist-test-slave-tm1g"
server_key: "198d147d6402acb1c2ec07495b786869"
server_key_iv: "81a071d36a95c07a09f86042080eec91"
server_key_version: "encryptionkey@0"
I20260807 03:09:58.529595 27391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "49b888fb6b624176a09ff790368af10d"
format_stamp: "Formatted at 2026-08-07 03:09:58 on dist-test-slave-tm1g"
server_key: "198d147d6402acb1c2ec07495b786869"
server_key_iv: "81a071d36a95c07a09f86042080eec91"
server_key_version: "encryptionkey@0"
I20260807 03:09:58.534766 27391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260807 03:09:58.537993 27405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:58.539304 27391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:09:58.539479 27391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "49b888fb6b624176a09ff790368af10d"
format_stamp: "Formatted at 2026-08-07 03:09:58 on dist-test-slave-tm1g"
server_key: "198d147d6402acb1c2ec07495b786869"
server_key_iv: "81a071d36a95c07a09f86042080eec91"
server_key_version: "encryptionkey@0"
I20260807 03:09:58.539695 27391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:58.559098 27391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:58.559916 27391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:58.560204 27391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:58.561192 27391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:58.562659 27391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:58.562791 27391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:58.562927 27391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:58.563071 27391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:58.596662 27391 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:40859
I20260807 03:09:58.596716 27523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:40859 every 8 connection(s)
I20260807 03:09:58.597960 27391 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:09:58.602468 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 27391
I20260807 03:09:58.602687 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:09:58.603510 17532 external_mini_cluster.cc:1428] Setting key 33a73e574e28869be8c62d6371524243
I20260807 03:09:58.606587 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37125
--builtin_ntp_servers=127.17.31.20:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 03:09:58.612195 27526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:58.600487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:45489 (local address 127.17.31.62:37125)
0807 03:09:58.600931 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:58.600936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:58.602047 (+ 1111us) server_negotiation.cc:407] Connection header received
0807 03:09:58.604575 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:58.604580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:58.604694 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:58.604811 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:58.606056 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:58.606941 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:58.609410 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:58.609801 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:58.609899 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:58.611469 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:58.611474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:58.611477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:58.611494 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:58.611535 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:58.611547 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:58.611620 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:58.611862 (+ 242us) server_negotiation.cc:299] Negotiation successful
0807 03:09:58.611940 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":248,"threads_started":1}
I20260807 03:09:58.613620 27527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:58.600961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:37125 (local address 127.17.31.1:45489)
0807 03:09:58.601832 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:58.601872 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:58.604260 (+ 2388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:58.605006 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:58.605013 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:58.605095 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:58.605868 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:58.605894 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:58.607080 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:58.607088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:58.609250 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:58.609267 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:58.610028 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:58.610037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:58.610378 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:58.611111 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:09:58.611116 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:58.611166 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:09:58.611269 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:58.611645 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:58.611658 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:58.611668 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 03:09:58.611851 (+ 183us) client_negotiation.cc:240] Negotiation successful
0807 03:09:58.612257 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":285,"threads_started":1}
I20260807 03:09:58.615226 27524 heartbeater.cc:347] Connected to a master server at 127.17.31.62:37125
I20260807 03:09:58.615526 27524 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:58.616451 27524 heartbeater.cc:511] Master 127.17.31.62:37125 requested a full tablet report, sending...
I20260807 03:09:58.618643 27331 ts_manager.cc:194] Registered new tserver with Master: 49b888fb6b624176a09ff790368af10d (127.17.31.1:40859)
I20260807 03:09:58.620873 27331 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.31.1:45489
W20260807 03:09:58.934569 27528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:58.934818 27528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:58.934899 27528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:58.941818 27528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:58.941903 27528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:58.941999 27528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:09:58.950901 27528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:58.952046 27528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:58.953711 27528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:58.966485 27537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:58.966533 27535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:58.966574 27534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:58.967322 27528 server_base.cc:1034] running on GCE node
I20260807 03:09:58.967864 27528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:58.968717 27528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:58.969959 27528 hybrid_clock.cc:648] HybridClock initialized: now 1786072198969907 us; error 38 us; skew 500 ppm
I20260807 03:09:58.970271 27528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:09:58.972576 27528 webserver.cc:460] Webserver started at http://127.17.31.2:42019/ using document root <none> and password file <none>
I20260807 03:09:58.973191 27528 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:58.973273 27528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:58.973552 27528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:58.976198 27528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "9d979bb4f04e4f83a3bd88168b0c952c"
format_stamp: "Formatted at 2026-08-07 03:09:58 on dist-test-slave-tm1g"
server_key: "836e6f0cc4ca054abed7c314b554465a"
server_key_iv: "87fca36911fd6b6dfdd87bdf36ebb861"
server_key_version: "encryptionkey@0"
I20260807 03:09:58.976915 27528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "9d979bb4f04e4f83a3bd88168b0c952c"
format_stamp: "Formatted at 2026-08-07 03:09:58 on dist-test-slave-tm1g"
server_key: "836e6f0cc4ca054abed7c314b554465a"
server_key_iv: "87fca36911fd6b6dfdd87bdf36ebb861"
server_key_version: "encryptionkey@0"
I20260807 03:09:58.981763 27528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 03:09:58.985049 27543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:58.986385 27528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 03:09:58.986569 27528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "9d979bb4f04e4f83a3bd88168b0c952c"
format_stamp: "Formatted at 2026-08-07 03:09:58 on dist-test-slave-tm1g"
server_key: "836e6f0cc4ca054abed7c314b554465a"
server_key_iv: "87fca36911fd6b6dfdd87bdf36ebb861"
server_key_version: "encryptionkey@0"
I20260807 03:09:58.986800 27528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:59.013475 27528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:59.014281 27528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:59.014607 27528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:59.015576 27528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:59.016963 27528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:59.017088 27528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:59.017204 27528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:59.017289 27528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:59.054090 27528 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:45115
I20260807 03:09:59.054171 27661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:45115 every 8 connection(s)
I20260807 03:09:59.055327 27528 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:09:59.065130 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 27528
I20260807 03:09:59.065351 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:09:59.065886 17532 external_mini_cluster.cc:1428] Setting key a9444526eee02f6094fde93e9f7e6c70
I20260807 03:09:59.067565 27526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:59.057827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:48513 (local address 127.17.31.62:37125)
0807 03:09:59.058015 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:59.058023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:59.059348 (+ 1325us) server_negotiation.cc:407] Connection header received
0807 03:09:59.060953 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:59.060957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:59.061020 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:59.061107 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:59.062022 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:59.062823 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:59.064392 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:59.064853 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:59.064956 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:59.066738 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:59.066743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:59.066746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:59.066767 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:59.066814 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:59.066825 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:59.066933 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:59.067142 (+ 209us) server_negotiation.cc:299] Negotiation successful
0807 03:09:59.067238 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 03:09:59.068651 27664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:59.058342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:37125 (local address 127.17.31.2:48513)
0807 03:09:59.059198 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:59.059236 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:59.060734 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:59.061281 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:59.061288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:59.061369 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:59.061887 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:59.061903 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:59.062957 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:59.062960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:59.064243 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:59.064255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:59.065069 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:59.065077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:59.065401 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:59.066177 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:09:59.066179 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:59.066214 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:09:59.066287 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:59.066932 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:59.066945 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:59.066954 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 03:09:59.067130 (+ 176us) client_negotiation.cc:240] Negotiation successful
0807 03:09:59.067498 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":272,"threads_started":1}
I20260807 03:09:59.068668 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37125
--builtin_ntp_servers=127.17.31.20:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 03:09:59.069988 27662 heartbeater.cc:347] Connected to a master server at 127.17.31.62:37125
I20260807 03:09:59.070302 27662 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:59.071192 27662 heartbeater.cc:511] Master 127.17.31.62:37125 requested a full tablet report, sending...
I20260807 03:09:59.072705 27331 ts_manager.cc:194] Registered new tserver with Master: 9d979bb4f04e4f83a3bd88168b0c952c (127.17.31.2:45115)
I20260807 03:09:59.073894 27331 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.31.2:48513
W20260807 03:09:59.395874 27665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:09:59.396134 27665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:09:59.396298 27665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:09:59.403335 27665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:09:59.403420 27665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:09:59.403614 27665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:09:59.412317 27665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:09:59.413537 27665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:09:59.415266 27665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:09:59.427892 27670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:59.427814 27673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:09:59.427933 27671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:09:59.428923 27665 server_base.cc:1034] running on GCE node
I20260807 03:09:59.429471 27665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:09:59.430377 27665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:09:59.431584 27665 hybrid_clock.cc:648] HybridClock initialized: now 1786072199431546 us; error 31 us; skew 500 ppm
I20260807 03:09:59.431885 27665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:09:59.433986 27665 webserver.cc:460] Webserver started at http://127.17.31.3:40083/ using document root <none> and password file <none>
I20260807 03:09:59.434723 27665 fs_manager.cc:359] Metadata directory not provided
I20260807 03:09:59.434841 27665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:09:59.435190 27665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:09:59.437821 27665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "6c9f6cbc545d469d9dda84973eee7b08"
format_stamp: "Formatted at 2026-08-07 03:09:59 on dist-test-slave-tm1g"
server_key: "afe5a1d92c6dd74028d19319b8c845ad"
server_key_iv: "d4827305572103152379de303e42ca2f"
server_key_version: "encryptionkey@0"
I20260807 03:09:59.438670 27665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "6c9f6cbc545d469d9dda84973eee7b08"
format_stamp: "Formatted at 2026-08-07 03:09:59 on dist-test-slave-tm1g"
server_key: "afe5a1d92c6dd74028d19319b8c845ad"
server_key_iv: "d4827305572103152379de303e42ca2f"
server_key_version: "encryptionkey@0"
I20260807 03:09:59.443619 27665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260807 03:09:59.446822 27679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:59.448024 27665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s
I20260807 03:09:59.448203 27665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "6c9f6cbc545d469d9dda84973eee7b08"
format_stamp: "Formatted at 2026-08-07 03:09:59 on dist-test-slave-tm1g"
server_key: "afe5a1d92c6dd74028d19319b8c845ad"
server_key_iv: "d4827305572103152379de303e42ca2f"
server_key_version: "encryptionkey@0"
I20260807 03:09:59.448424 27665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:09:59.462492 27665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:09:59.463243 27665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:09:59.463529 27665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:09:59.464423 27665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:09:59.465831 27665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:09:59.465924 27665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:59.466003 27665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:09:59.466058 27665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:09:59.498121 27665 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:41589
I20260807 03:09:59.498204 27798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:41589 every 8 connection(s)
I20260807 03:09:59.499475 27665 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:09:59.503736 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 27665
I20260807 03:09:59.503921 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:09:59.504343 17532 external_mini_cluster.cc:1428] Setting key 85cf8bf30647fd6a02fbb93392e26f87
I20260807 03:09:59.511355 27526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:59.501856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:33429 (local address 127.17.31.62:37125)
0807 03:09:59.502036 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:59.502041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:59.503361 (+ 1320us) server_negotiation.cc:407] Connection header received
0807 03:09:59.505218 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:59.505225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:59.505322 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:59.505441 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:59.506447 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:59.507242 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:59.508735 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:59.509172 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:59.509303 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:59.510552 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:59.510557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:59.510560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:59.510577 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:59.510609 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:59.510620 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:59.510719 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:59.510935 (+ 216us) server_negotiation.cc:299] Negotiation successful
0807 03:09:59.511028 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 03:09:59.512226 27802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:59.502375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:37125 (local address 127.17.31.3:33429)
0807 03:09:59.503218 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0807 03:09:59.503252 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 03:09:59.504931 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:09:59.505654 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:09:59.505662 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:09:59.505740 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:09:59.506254 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:59.506269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:59.507391 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:59.507397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:59.508589 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:09:59.508600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:09:59.509404 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:09:59.509412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:09:59.509623 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:59.510214 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:09:59.510217 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:09:59.510251 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:09:59.510364 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:09:59.510715 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:09:59.510726 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:09:59.510736 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 03:09:59.510901 (+ 165us) client_negotiation.cc:240] Negotiation successful
0807 03:09:59.511242 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":260,"threads_started":1}
I20260807 03:09:59.513417 27800 heartbeater.cc:347] Connected to a master server at 127.17.31.62:37125
I20260807 03:09:59.513721 27800 heartbeater.cc:464] Registering TS with master...
I20260807 03:09:59.514504 27800 heartbeater.cc:511] Master 127.17.31.62:37125 requested a full tablet report, sending...
I20260807 03:09:59.515713 27331 ts_manager.cc:194] Registered new tserver with Master: 6c9f6cbc545d469d9dda84973eee7b08 (127.17.31.3:41589)
I20260807 03:09:59.516525 27331 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.31.3:33429
I20260807 03:09:59.519218 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 03:09:59.525161 27526 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:09:59.523574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42166 (local address 127.17.31.62:37125)
0807 03:09:59.523728 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:59.523734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:59.524149 (+ 415us) server_negotiation.cc:407] Connection header received
0807 03:09:59.524336 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:59.524341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:59.524420 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:59.524523 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:59.524929 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42166: BlockingRecv error: recv got EOF from 127.0.0.1:42166 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 03:09:59.525682 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 27391
I20260807 03:09:59.532827 27809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:59.526689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.17.31.1:40859)
0807 03:09:59.527194 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:59.527200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:59.527223 (+ 23us) server_negotiation.cc:407] Connection header received
0807 03:09:59.527274 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:59.527279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:59.527437 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:59.527553 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:59.527913 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:59.528883 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:59.530035 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:59.530590 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:59.530682 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:59.531577 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:59.531584 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:59.531594 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:59.532108 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:59.532147 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:59.532164 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:59.532247 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:59.532350 (+ 103us) server_negotiation.cc:299] Negotiation successful
0807 03:09:59.532488 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":265,"threads_started":1}
I20260807 03:09:59.533896 27519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:59.615111 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 27391
I20260807 03:09:59.633198 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 27528
I20260807 03:09:59.640836 27811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:59.634109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.17.31.2:45115)
0807 03:09:59.634720 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:59.634727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:59.634753 (+ 26us) server_negotiation.cc:407] Connection header received
0807 03:09:59.634814 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:59.634820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:59.635032 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:59.635158 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:59.635496 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:59.636513 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:59.637651 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:59.638183 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:59.638275 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:59.639447 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:59.639454 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:59.639466 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:59.640035 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:59.640078 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:59.640097 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:59.640212 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:59.640361 (+ 149us) server_negotiation.cc:299] Negotiation successful
0807 03:09:59.640521 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":343,"threads_started":1}
I20260807 03:09:59.641969 27657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:59.722985 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 27528
I20260807 03:09:59.741201 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 27665
I20260807 03:09:59.751076 27813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:09:59.742467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39104 (local address 127.17.31.3:41589)
0807 03:09:59.743179 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0807 03:09:59.743187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:09:59.743218 (+ 31us) server_negotiation.cc:407] Connection header received
0807 03:09:59.743280 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:09:59.743286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:09:59.743516 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:09:59.743677 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:09:59.744199 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:59.745821 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:59.747644 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:09:59.748302 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:09:59.748452 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:09:59.749520 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:09:59.749529 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:09:59.749555 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:09:59.750159 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:09:59.750209 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:09:59.750229 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:09:59.750392 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 03:09:59.750556 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 03:09:59.750699 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":385,"threads_started":1}
I20260807 03:09:59.752344 27794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:59.834172 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 27665
I20260807 03:09:59.850569 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 27299
I20260807 03:09:59.851763 27362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:09:59.953305 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 27299
2026-08-07T03:09:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2430 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-07T03:10:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:10:00Z Disabled control of system clock
I20260807 03:10:00.028802 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:42401
--webserver_interface=127.17.31.62
--webserver_port=0
--builtin_ntp_servers=127.17.31.20:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:42401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 03:10:00.352977 27820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:10:00.353252 27820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:10:00.353334 27820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:10:00.360172 27820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:10:00.360258 27820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:10:00.360298 27820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:10:00.360344 27820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:10:00.360379 27820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:10:00.368902 27820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:42401
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:42401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:10:00.370060 27820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:10:00.371724 27820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:10:00.383307 27826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:00.383270 27828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:00.383509 27825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:10:00.384289 27820 server_base.cc:1034] running on GCE node
I20260807 03:10:00.385020 27820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:10:00.386257 27820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:10:00.387512 27820 hybrid_clock.cc:648] HybridClock initialized: now 1786072200387472 us; error 33 us; skew 500 ppm
I20260807 03:10:00.387840 27820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:10:00.390017 27820 webserver.cc:460] Webserver started at http://127.17.31.62:42067/ using document root <none> and password file <none>
I20260807 03:10:00.390767 27820 fs_manager.cc:359] Metadata directory not provided
I20260807 03:10:00.390916 27820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:10:00.391244 27820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:10:00.395776 27820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "3c8023813820469e988e58466489b8ed"
format_stamp: "Formatted at 2026-08-07 03:10:00 on dist-test-slave-tm1g"
server_key: "84ac9de3efa413ace3d7438941e19301"
server_key_iv: "f879c95bba310c70c1fb77e09983de56"
server_key_version: "encryptionkey@0"
I20260807 03:10:00.396564 27820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "3c8023813820469e988e58466489b8ed"
format_stamp: "Formatted at 2026-08-07 03:10:00 on dist-test-slave-tm1g"
server_key: "84ac9de3efa413ace3d7438941e19301"
server_key_iv: "f879c95bba310c70c1fb77e09983de56"
server_key_version: "encryptionkey@0"
I20260807 03:10:00.401615 27820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260807 03:10:00.404982 27834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:00.406517 27820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 03:10:00.406687 27820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "3c8023813820469e988e58466489b8ed"
format_stamp: "Formatted at 2026-08-07 03:10:00 on dist-test-slave-tm1g"
server_key: "84ac9de3efa413ace3d7438941e19301"
server_key_iv: "f879c95bba310c70c1fb77e09983de56"
server_key_version: "encryptionkey@0"
I20260807 03:10:00.406908 27820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:10:00.412592 27820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:10:00.413331 27820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:10:00.413623 27820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:10:00.432789 27820 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:42401
I20260807 03:10:00.432880 27889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:42401 every 8 connection(s)
I20260807 03:10:00.434075 27820 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:10:00.437922 27890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:10:00.438199 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 27820
I20260807 03:10:00.438412 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:10:00.439061 17532 external_mini_cluster.cc:1428] Setting key ae86b7c9c58e3986c9fd69a36bcbb92b
I20260807 03:10:00.447947 27893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:00.441049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54930 (local address 127.17.31.62:42401)
0807 03:10:00.442123 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:00.442133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:00.442177 (+ 44us) server_negotiation.cc:407] Connection header received
0807 03:10:00.444117 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:00.444145 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:00.444563 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:00.444939 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:00.446294 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:00.446370 (+ 76us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:00.446391 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:10:00.446416 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:00.446480 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:10:00.446497 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:00.446666 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:00.446810 (+ 144us) server_negotiation.cc:299] Negotiation successful
0807 03:10:00.447084 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":412,"threads_started":1}
I20260807 03:10:00.451043 27890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed: Bootstrap starting.
I20260807 03:10:00.453826 27890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed: Neither blocks nor log segments found. Creating new log.
I20260807 03:10:00.454842 27890 log.cc:824] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed: Log is configured to *not* fsync() on all Append() calls
I20260807 03:10:00.457516 27890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed: No bootstrap required, opened a new log
I20260807 03:10:00.464211 27890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8023813820469e988e58466489b8ed" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42401 } }
I20260807 03:10:00.464519 27890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:10:00.464591 27890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8023813820469e988e58466489b8ed, State: Initialized, Role: FOLLOWER
I20260807 03:10:00.465031 27890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8023813820469e988e58466489b8ed" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42401 } }
I20260807 03:10:00.465219 27890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:10:00.465327 27890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:10:00.465453 27890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:10:00.467512 27890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8023813820469e988e58466489b8ed" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42401 } }
I20260807 03:10:00.467912 27890 leader_election.cc:302] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c8023813820469e988e58466489b8ed; no voters:
I20260807 03:10:00.468554 27890 leader_election.cc:288] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:10:00.468711 27895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:10:00.469444 27895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [term 1 LEADER]: Becoming Leader. State: Replica: 3c8023813820469e988e58466489b8ed, State: Running, Role: LEADER
I20260807 03:10:00.470013 27895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8023813820469e988e58466489b8ed" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42401 } }
I20260807 03:10:00.470628 27890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:10:00.472530 27896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c8023813820469e988e58466489b8ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8023813820469e988e58466489b8ed" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42401 } } }
I20260807 03:10:00.472829 27896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [sys.catalog]: This master's current role is: LEADER
I20260807 03:10:00.474434 27901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:10:00.474416 27897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c8023813820469e988e58466489b8ed. Latest consensus state: current_term: 1 leader_uuid: "3c8023813820469e988e58466489b8ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8023813820469e988e58466489b8ed" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42401 } } }
I20260807 03:10:00.474784 27897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed [sys.catalog]: This master's current role is: LEADER
I20260807 03:10:00.480085 27901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:10:00.488569 27901 catalog_manager.cc:1347] Generated new cluster ID: 4c9708beb38b4e979989ab3da0895e92
I20260807 03:10:00.488684 27901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:10:00.504789 27901 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:10:00.505216 27901 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:10:00.510270 27901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c8023813820469e988e58466489b8ed: Generated new TSK 0
I20260807 03:10:00.511104 27901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:10:00.521520 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42401
--builtin_ntp_servers=127.17.31.20:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 03:10:00.858074 27917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:10:00.858363 27917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:10:00.858453 27917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:10:00.865442 27917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:10:00.865535 27917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:10:00.865643 27917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:10:00.874773 27917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:10:00.876205 27917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:10:00.878075 27917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:10:00.890904 27922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:00.890861 27923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:00.890954 27925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:10:00.891664 27917 server_base.cc:1034] running on GCE node
I20260807 03:10:00.892288 27917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:10:00.893194 27917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:10:00.894461 27917 hybrid_clock.cc:648] HybridClock initialized: now 1786072200894421 us; error 51 us; skew 500 ppm
I20260807 03:10:00.894795 27917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:10:00.897078 27917 webserver.cc:460] Webserver started at http://127.17.31.1:36873/ using document root <none> and password file <none>
I20260807 03:10:00.897771 27917 fs_manager.cc:359] Metadata directory not provided
I20260807 03:10:00.897912 27917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:10:00.898236 27917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:10:00.901288 27917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "33c30263c8b1466680336f45ea612942"
format_stamp: "Formatted at 2026-08-07 03:10:00 on dist-test-slave-tm1g"
server_key: "6f2b046def844308c8b5de58f63ca0a9"
server_key_iv: "ba24bd208099359718b48885f1f66b21"
server_key_version: "encryptionkey@0"
I20260807 03:10:00.902119 27917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "33c30263c8b1466680336f45ea612942"
format_stamp: "Formatted at 2026-08-07 03:10:00 on dist-test-slave-tm1g"
server_key: "6f2b046def844308c8b5de58f63ca0a9"
server_key_iv: "ba24bd208099359718b48885f1f66b21"
server_key_version: "encryptionkey@0"
I20260807 03:10:00.907120 27917 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260807 03:10:00.910470 27932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:00.911840 27917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 03:10:00.912009 27917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "33c30263c8b1466680336f45ea612942"
format_stamp: "Formatted at 2026-08-07 03:10:00 on dist-test-slave-tm1g"
server_key: "6f2b046def844308c8b5de58f63ca0a9"
server_key_iv: "ba24bd208099359718b48885f1f66b21"
server_key_version: "encryptionkey@0"
I20260807 03:10:00.912225 27917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:10:00.917779 27917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:10:00.918586 27917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:10:00.918850 27917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:10:00.919719 27917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:10:00.921042 27917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:10:00.921173 27917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:00.921296 27917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:10:00.921392 27917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:00.954278 27917 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:33847
I20260807 03:10:00.954399 28050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:33847 every 8 connection(s)
I20260807 03:10:00.955657 27917 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:10:00.959363 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 27917
I20260807 03:10:00.959574 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:10:00.960230 17532 external_mini_cluster.cc:1428] Setting key 45012e47c5ae6922e29ff472dc168a83
I20260807 03:10:00.963581 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42401
--builtin_ntp_servers=127.17.31.20:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 03:10:00.964424 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:00.958172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:60705 (local address 127.17.31.62:42401)
0807 03:10:00.958641 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:00.958647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:00.960002 (+ 1355us) server_negotiation.cc:407] Connection header received
0807 03:10:00.962032 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:00.962037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:00.962094 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:00.962230 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:00.963712 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:00.963719 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:00.963726 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:10:00.963752 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:00.963813 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:10:00.963832 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:00.963929 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:00.964148 (+ 219us) server_negotiation.cc:299] Negotiation successful
0807 03:10:00.964196 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":237,"threads_started":1}
I20260807 03:10:00.965703 28055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:00.958780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:42401 (local address 127.17.31.1:60705)
0807 03:10:00.959849 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:00.959900 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:00.961768 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:00.962494 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:00.962507 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:00.962622 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:10:00.963330 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:10:00.963335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:10:00.963391 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:10:00.963505 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:10:00.963933 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:10:00.963946 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:10:00.963957 (+ 11us) client_negotiation.cc:769] Sending connection context
0807 03:10:00.964225 (+ 268us) client_negotiation.cc:240] Negotiation successful
0807 03:10:00.964481 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":361,"threads_started":1}
I20260807 03:10:00.967087 28051 heartbeater.cc:347] Connected to a master server at 127.17.31.62:42401
I20260807 03:10:00.967362 28051 heartbeater.cc:464] Registering TS with master...
I20260807 03:10:00.967921 28051 heartbeater.cc:511] Master 127.17.31.62:42401 requested a full tablet report, sending...
I20260807 03:10:00.969834 27851 ts_manager.cc:194] Registered new tserver with Master: 33c30263c8b1466680336f45ea612942 (127.17.31.1:33847)
W20260807 03:10:01.288179 28056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:10:01.288434 28056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:10:01.288605 28056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:10:01.295678 28056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:10:01.295768 28056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:10:01.295917 28056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:10:01.304522 28056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:10:01.305799 28056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:10:01.307525 28056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:10:01.320480 28064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:01.320581 28061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:01.320554 28062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:10:01.321257 28056 server_base.cc:1034] running on GCE node
I20260807 03:10:01.321872 28056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:10:01.322811 28056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:10:01.324005 28056 hybrid_clock.cc:648] HybridClock initialized: now 1786072201323969 us; error 31 us; skew 500 ppm
I20260807 03:10:01.324414 28056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:10:01.326520 28056 webserver.cc:460] Webserver started at http://127.17.31.2:38665/ using document root <none> and password file <none>
I20260807 03:10:01.327171 28056 fs_manager.cc:359] Metadata directory not provided
I20260807 03:10:01.327281 28056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:10:01.327622 28056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:10:01.330404 28056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "d4373277e5a5419f887b37343d5cf896"
format_stamp: "Formatted at 2026-08-07 03:10:01 on dist-test-slave-tm1g"
server_key: "6dac87bb4f8265d5e48947d95bffd35c"
server_key_iv: "46a17cd4bf043114d4c137b69e8b7e22"
server_key_version: "encryptionkey@0"
I20260807 03:10:01.331154 28056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "d4373277e5a5419f887b37343d5cf896"
format_stamp: "Formatted at 2026-08-07 03:10:01 on dist-test-slave-tm1g"
server_key: "6dac87bb4f8265d5e48947d95bffd35c"
server_key_iv: "46a17cd4bf043114d4c137b69e8b7e22"
server_key_version: "encryptionkey@0"
I20260807 03:10:01.335990 28056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 03:10:01.339316 28071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:01.340523 28056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:10:01.340684 28056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "d4373277e5a5419f887b37343d5cf896"
format_stamp: "Formatted at 2026-08-07 03:10:01 on dist-test-slave-tm1g"
server_key: "6dac87bb4f8265d5e48947d95bffd35c"
server_key_iv: "46a17cd4bf043114d4c137b69e8b7e22"
server_key_version: "encryptionkey@0"
I20260807 03:10:01.340898 28056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:10:01.346208 28056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:10:01.346952 28056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:10:01.347220 28056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:10:01.348083 28056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:10:01.349381 28056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:10:01.349476 28056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:01.349629 28056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:10:01.349723 28056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:01.382239 28056 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:41845
I20260807 03:10:01.382350 28188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:41845 every 8 connection(s)
I20260807 03:10:01.383608 28056 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:10:01.388958 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 28056
I20260807 03:10:01.389124 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:10:01.389585 17532 external_mini_cluster.cc:1428] Setting key 4786ad9165a84fffcea36df371d5f976
I20260807 03:10:01.391819 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:01.386036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:32797 (local address 127.17.31.62:42401)
0807 03:10:01.386259 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:01.386265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:01.387680 (+ 1415us) server_negotiation.cc:407] Connection header received
0807 03:10:01.389350 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:01.389355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:01.389443 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:01.389564 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:01.391020 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:01.391025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:01.391028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:10:01.391043 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:01.391117 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:10:01.391138 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:01.391230 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:01.391475 (+ 245us) server_negotiation.cc:299] Negotiation successful
0807 03:10:01.391542 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 03:10:01.392542 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42401
--builtin_ntp_servers=127.17.31.20:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 03:10:01.392697 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:01.386644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:42401 (local address 127.17.31.2:32797)
0807 03:10:01.387529 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:01.387563 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:01.389043 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:01.389780 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:01.389788 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:01.389871 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:10:01.390689 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:10:01.390692 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:10:01.390738 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:10:01.390839 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:10:01.391242 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:10:01.391252 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:10:01.391259 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 03:10:01.391555 (+ 296us) client_negotiation.cc:240] Negotiation successful
0807 03:10:01.391789 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":279,"threads_started":1}
I20260807 03:10:01.394045 28189 heartbeater.cc:347] Connected to a master server at 127.17.31.62:42401
I20260807 03:10:01.394364 28189 heartbeater.cc:464] Registering TS with master...
I20260807 03:10:01.394905 28189 heartbeater.cc:511] Master 127.17.31.62:42401 requested a full tablet report, sending...
I20260807 03:10:01.396199 27851 ts_manager.cc:194] Registered new tserver with Master: d4373277e5a5419f887b37343d5cf896 (127.17.31.2:41845)
W20260807 03:10:01.717417 28192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:10:01.717664 28192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:10:01.717749 28192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:10:01.724604 28192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:10:01.724686 28192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:10:01.724786 28192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:10:01.733877 28192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:42401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:10:01.735605 28192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:10:01.737324 28192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:10:01.754024 28199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:01.753969 28198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:01.754062 28201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:10:01.754916 28192 server_base.cc:1034] running on GCE node
I20260807 03:10:01.755501 28192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:10:01.756372 28192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:10:01.757593 28192 hybrid_clock.cc:648] HybridClock initialized: now 1786072201757608 us; error 116 us; skew 500 ppm
I20260807 03:10:01.757896 28192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:10:01.760027 28192 webserver.cc:460] Webserver started at http://127.17.31.3:41997/ using document root <none> and password file <none>
I20260807 03:10:01.760695 28192 fs_manager.cc:359] Metadata directory not provided
I20260807 03:10:01.760833 28192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:10:01.761138 28192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:10:01.763912 28192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "af378b9050a94a94a8cdbf38812770ea"
format_stamp: "Formatted at 2026-08-07 03:10:01 on dist-test-slave-tm1g"
server_key: "88d3f07e583fea846f7f1594e9599b37"
server_key_iv: "28acf315fc4366062d81d64351414e29"
server_key_version: "encryptionkey@0"
I20260807 03:10:01.764686 28192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "af378b9050a94a94a8cdbf38812770ea"
format_stamp: "Formatted at 2026-08-07 03:10:01 on dist-test-slave-tm1g"
server_key: "88d3f07e583fea846f7f1594e9599b37"
server_key_iv: "28acf315fc4366062d81d64351414e29"
server_key_version: "encryptionkey@0"
I20260807 03:10:01.769544 28192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260807 03:10:01.772827 28207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:01.773983 28192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 03:10:01.774150 28192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "af378b9050a94a94a8cdbf38812770ea"
format_stamp: "Formatted at 2026-08-07 03:10:01 on dist-test-slave-tm1g"
server_key: "88d3f07e583fea846f7f1594e9599b37"
server_key_iv: "28acf315fc4366062d81d64351414e29"
server_key_version: "encryptionkey@0"
I20260807 03:10:01.774384 28192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:10:01.779831 28192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:10:01.780678 28192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:10:01.780936 28192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:10:01.781726 28192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:10:01.783041 28192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:10:01.783165 28192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:01.783283 28192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:10:01.783370 28192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:01.815874 28192 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:41457
I20260807 03:10:01.815927 28329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:41457 every 8 connection(s)
I20260807 03:10:01.817155 28192 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:10:01.817734 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 28192
I20260807 03:10:01.817924 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:10:01.818616 17532 external_mini_cluster.cc:1428] Setting key a2f9da547215c0ae45553fbec373b11d
I20260807 03:10:01.825417 28054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:01.820064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:38181 (local address 127.17.31.62:42401)
0807 03:10:01.820256 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:01.820262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:01.821850 (+ 1588us) server_negotiation.cc:407] Connection header received
0807 03:10:01.823578 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:01.823582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:01.823632 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:01.823709 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:01.824817 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:01.824821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:01.824825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:10:01.824839 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:01.824869 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:10:01.824881 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:01.824975 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:01.825134 (+ 159us) server_negotiation.cc:299] Negotiation successful
0807 03:10:01.825183 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 03:10:01.826211 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:01.820579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:42401 (local address 127.17.31.3:38181)
0807 03:10:01.821677 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:01.821733 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:01.823341 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:01.823893 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:01.823900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:01.823979 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:10:01.824562 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:10:01.824565 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:10:01.824598 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:10:01.824676 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:10:01.824965 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:10:01.824972 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:10:01.824978 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 03:10:01.825176 (+ 198us) client_negotiation.cc:240] Negotiation successful
0807 03:10:01.825368 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":315,"threads_started":1}
I20260807 03:10:01.827521 28330 heartbeater.cc:347] Connected to a master server at 127.17.31.62:42401
I20260807 03:10:01.827759 28330 heartbeater.cc:464] Registering TS with master...
I20260807 03:10:01.828243 28330 heartbeater.cc:511] Master 127.17.31.62:42401 requested a full tablet report, sending...
I20260807 03:10:01.829494 27851 ts_manager.cc:194] Registered new tserver with Master: af378b9050a94a94a8cdbf38812770ea (127.17.31.3:41457)
I20260807 03:10:01.834101 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 03:10:01.840034 28054 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:10:01.838506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.17.31.62:42401)
0807 03:10:01.838676 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:01.838684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:01.839313 (+ 629us) server_negotiation.cc:407] Connection header received
0807 03:10:01.839412 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:01.839415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:01.839459 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:01.839534 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:01.839792 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54944: BlockingRecv error: recv got EOF from 127.0.0.1:54944 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 03:10:01.840675 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 27917
I20260807 03:10:01.844210 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:01.841600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.17.31.1:33847)
0807 03:10:01.842077 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:01.842083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:01.842104 (+ 21us) server_negotiation.cc:407] Connection header received
0807 03:10:01.842154 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:01.842159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:01.842401 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:01.842564 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:01.843520 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:01.843526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:01.843535 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:10:01.843551 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:01.843586 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:10:01.843599 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:01.843739 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:01.843843 (+ 104us) server_negotiation.cc:299] Negotiation successful
0807 03:10:01.843942 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":228,"threads_started":1}
I20260807 03:10:01.845271 28046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:10:01.924738 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 27917
I20260807 03:10:01.942387 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 28056
I20260807 03:10:01.945979 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:01.943307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57634 (local address 127.17.31.2:41845)
0807 03:10:01.943870 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:01.943877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:01.943903 (+ 26us) server_negotiation.cc:407] Connection header received
0807 03:10:01.943979 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:01.943985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:01.944224 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:01.944344 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:01.945370 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:01.945377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:01.945386 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:10:01.945399 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:01.945437 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:10:01.945451 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:01.945555 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:01.945657 (+ 102us) server_negotiation.cc:299] Negotiation successful
0807 03:10:01.945733 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":284,"threads_started":1}
I20260807 03:10:01.947290 28184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:10:02.028910 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 28056
I20260807 03:10:02.046635 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 28192
I20260807 03:10:02.050551 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:02.047711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.17.31.3:41457)
0807 03:10:02.048450 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:02.048464 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:02.048506 (+ 42us) server_negotiation.cc:407] Connection header received
0807 03:10:02.048571 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:02.048578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:02.048805 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:02.048936 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:02.049788 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:02.049794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:02.049806 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:10:02.049816 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:02.049858 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:10:02.049876 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:02.050006 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:02.050053 (+ 47us) server_negotiation.cc:299] Negotiation successful
0807 03:10:02.050171 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":287,"threads_started":1}
I20260807 03:10:02.051779 28325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:10:02.132890 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 28192
I20260807 03:10:02.150024 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 27820
I20260807 03:10:02.151484 27884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:10:02.257689 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 27820
2026-08-07T03:10:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2304 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:10:02 dist-test-slave-tm1g krb5kdc[28349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:10:02 dist-test-slave-tm1g krb5kdc[28349](info): set up 2 sockets
Aug 07 03:10:02 dist-test-slave-tm1g krb5kdc[28349](info): commencing operation
krb5kdc: starting...
W20260807 03:10:03.376696 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:10:03 dist-test-slave-tm1g krb5kdc[28349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:10:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:10:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:10:03.602011 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:41731
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:34329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:41731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:10:03.927702 28366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:10:03.928018 28366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:10:03.928186 28366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:10:03.935283 28366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:10:03.935410 28366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:10:03.935518 28366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:10:03.935623 28366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:10:03.935710 28366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:10:03.945685 28366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:34329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:41731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:41731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:10:03.946977 28366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:10:03.948324 28366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:10:03.959838 28371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:03.959867 28374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:03.959991 28372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:10:03.963611 28366 server_base.cc:1034] running on GCE node
I20260807 03:10:03.964419 28366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:10:03.965746 28366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:10:03.967031 28366 hybrid_clock.cc:648] HybridClock initialized: now 1786072203966968 us; error 31 us; skew 500 ppm
I20260807 03:10:03.967410 28366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:10:03 dist-test-slave-tm1g krb5kdc[28349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072203, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:10:03.970500 28366 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:10:03.971921 28366 webserver.cc:460] Webserver started at http://127.17.31.62:45285/ using document root <none> and password file <none>
I20260807 03:10:03.972645 28366 fs_manager.cc:359] Metadata directory not provided
I20260807 03:10:03.972807 28366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:10:03.973122 28366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:10:03.976053 28366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "4390780bf82e45c0896fdeffebd2354e"
format_stamp: "Formatted at 2026-08-07 03:10:03 on dist-test-slave-tm1g"
server_key: "1f51a14db8c8dfc54a95501562740b5b"
server_key_iv: "c242c44c47f8c2dc730a34e46b409824"
server_key_version: "encryptionkey@0"
I20260807 03:10:03.976846 28366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "4390780bf82e45c0896fdeffebd2354e"
format_stamp: "Formatted at 2026-08-07 03:10:03 on dist-test-slave-tm1g"
server_key: "1f51a14db8c8dfc54a95501562740b5b"
server_key_iv: "c242c44c47f8c2dc730a34e46b409824"
server_key_version: "encryptionkey@0"
I20260807 03:10:03.981751 28366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:10:03.985002 28381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:03.986277 28366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:10:03.986534 28366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "4390780bf82e45c0896fdeffebd2354e"
format_stamp: "Formatted at 2026-08-07 03:10:03 on dist-test-slave-tm1g"
server_key: "1f51a14db8c8dfc54a95501562740b5b"
server_key_iv: "c242c44c47f8c2dc730a34e46b409824"
server_key_version: "encryptionkey@0"
I20260807 03:10:03.986724 28366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:10:04.000809 28366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:10:04.004999 28366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:10:04.005300 28366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:10:04.023641 28366 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:41731
I20260807 03:10:04.023676 28436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:41731 every 8 connection(s)
I20260807 03:10:04.024905 28366 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:10:04.028473 28437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:10:04.032366 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 28366
I20260807 03:10:04.032606 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:10:04.033424 17532 external_mini_cluster.cc:1428] Setting key 357b8b6792e2f5ef60bf7a3f485e2171
I20260807 03:10:04.040017 28437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e: Bootstrap starting.
I20260807 03:10:04.043533 28437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e: Neither blocks nor log segments found. Creating new log.
I20260807 03:10:04.044644 28437 log.cc:824] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e: Log is configured to *not* fsync() on all Append() calls
I20260807 03:10:04.047382 28437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e: No bootstrap required, opened a new log
Aug 07 03:10:04 dist-test-slave-tm1g krb5kdc[28349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:10:04.055531 28437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4390780bf82e45c0896fdeffebd2354e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41731 } }
I20260807 03:10:04.055975 28437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:10:04.056092 28437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4390780bf82e45c0896fdeffebd2354e, State: Initialized, Role: FOLLOWER
I20260807 03:10:04.056715 28437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4390780bf82e45c0896fdeffebd2354e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41731 } }
I20260807 03:10:04.056942 28437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:10:04.057096 28437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:10:04.057293 28437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:10:04.059523 28437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4390780bf82e45c0896fdeffebd2354e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41731 } }
I20260807 03:10:04.059989 28437 leader_election.cc:302] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4390780bf82e45c0896fdeffebd2354e; no voters:
I20260807 03:10:04.061447 28437 leader_election.cc:288] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:10:04.061528 28442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:10:04.062587 28440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:04.035521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36046 (local address 127.17.31.62:41731)
0807 03:10:04.036294 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:04.036305 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:04.036341 (+ 36us) server_negotiation.cc:407] Connection header received
0807 03:10:04.038581 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:04.038601 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:04.038942 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:04.039234 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:04.041113 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:04.042465 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:04.044260 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:04.044957 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:04.045129 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:04.054362 (+ 9233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:04.054378 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:04.054391 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:10:04.054450 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:04.057819 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:04.058566 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:04.058574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:04.058581 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:04.058753 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:04.059134 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:04.059139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:04.059142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:04.059683 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:04.060023 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:04.060459 (+ 436us) server_negotiation.cc:299] Negotiation successful
0807 03:10:04.060982 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":398,"threads_started":1}
I20260807 03:10:04.063565 28437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:10:04.066672 28442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [term 1 LEADER]: Becoming Leader. State: Replica: 4390780bf82e45c0896fdeffebd2354e, State: Running, Role: LEADER
I20260807 03:10:04.067188 28442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4390780bf82e45c0896fdeffebd2354e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41731 } }
I20260807 03:10:04.070294 28445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4390780bf82e45c0896fdeffebd2354e. Latest consensus state: current_term: 1 leader_uuid: "4390780bf82e45c0896fdeffebd2354e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4390780bf82e45c0896fdeffebd2354e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41731 } } }
I20260807 03:10:04.070693 28445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [sys.catalog]: This master's current role is: LEADER
I20260807 03:10:04.070405 28443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4390780bf82e45c0896fdeffebd2354e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4390780bf82e45c0896fdeffebd2354e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41731 } } }
I20260807 03:10:04.070919 28443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e [sys.catalog]: This master's current role is: LEADER
W20260807 03:10:04.075757 28457 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:10:04.075886 28457 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:10:04.076043 28458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:10:04.078061 28458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:10:04.086494 28458 catalog_manager.cc:1347] Generated new cluster ID: e39d2ee0af3843638a8c64833ff3b9f7
I20260807 03:10:04.086614 28458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:10:04.108158 28458 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:10:04.109386 28458 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:10:04.115904 28458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e: Generated new TSK 0
I20260807 03:10:04.116762 28458 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:10:04.181424 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:41731
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:34329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:10:04.512495 28464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:10:04.512753 28464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:10:04.512903 28464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:10:04.519788 28464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:10:04.519881 28464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:10:04.520043 28464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:10:04.529217 28464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:34329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:41731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:10:04.530491 28464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:10:04.531770 28464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:10:04.544257 28469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:04.544281 28470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:04.544204 28472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:10:04.545112 28464 server_base.cc:1034] running on GCE node
I20260807 03:10:04.545684 28464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:10:04.546561 28464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:10:04.547758 28464 hybrid_clock.cc:648] HybridClock initialized: now 1786072204547724 us; error 25 us; skew 500 ppm
I20260807 03:10:04.548022 28464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:10:04 dist-test-slave-tm1g krb5kdc[28349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072204, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:10:04.551146 28464 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:10:04.552445 28464 webserver.cc:460] Webserver started at http://127.17.31.1:44977/ using document root <none> and password file <none>
I20260807 03:10:04.553094 28464 fs_manager.cc:359] Metadata directory not provided
I20260807 03:10:04.553232 28464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:10:04.553542 28464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:10:04.556236 28464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "4df4ec0cae9c428dbb3481243b52d9af"
format_stamp: "Formatted at 2026-08-07 03:10:04 on dist-test-slave-tm1g"
server_key: "39bc15770ca5569d9019a3b0308b4f8b"
server_key_iv: "ec6374646c765a1324a4e89c2ef8f847"
server_key_version: "encryptionkey@0"
I20260807 03:10:04.556972 28464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "4df4ec0cae9c428dbb3481243b52d9af"
format_stamp: "Formatted at 2026-08-07 03:10:04 on dist-test-slave-tm1g"
server_key: "39bc15770ca5569d9019a3b0308b4f8b"
server_key_iv: "ec6374646c765a1324a4e89c2ef8f847"
server_key_version: "encryptionkey@0"
I20260807 03:10:04.561980 28464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 03:10:04.565594 28479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:04.567086 28464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:10:04.567271 28464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "4df4ec0cae9c428dbb3481243b52d9af"
format_stamp: "Formatted at 2026-08-07 03:10:04 on dist-test-slave-tm1g"
server_key: "39bc15770ca5569d9019a3b0308b4f8b"
server_key_iv: "ec6374646c765a1324a4e89c2ef8f847"
server_key_version: "encryptionkey@0"
I20260807 03:10:04.567462 28464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:10:04.584723 28464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:10:04.588492 28464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:10:04.588734 28464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:10:04.589586 28464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:10:04.591089 28464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:10:04.591202 28464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:04.591302 28464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:10:04.591352 28464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:04.624275 28464 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:42437
I20260807 03:10:04.624377 28600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:42437 every 8 connection(s)
I20260807 03:10:04.625613 28464 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:10:04.634931 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 28464
I20260807 03:10:04.635143 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:10:04.635913 17532 external_mini_cluster.cc:1428] Setting key 13963f5d268f7cb7ba33899a1aa165a1
Aug 07 03:10:04 dist-test-slave-tm1g krb5kdc[28349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072204, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:10:04.645675 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:04.628131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:48437 (local address 127.17.31.62:41731)
0807 03:10:04.628575 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:04.628581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:04.629558 (+ 977us) server_negotiation.cc:407] Connection header received
0807 03:10:04.631339 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:04.631344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:04.631414 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:04.631514 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:04.632898 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:04.633611 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:04.635614 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:04.636066 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:04.636161 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:04.639984 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:04.639989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:04.639993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:10:04.640023 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:04.642703 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:04.643392 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:04.643399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:04.643403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:04.643524 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:04.643977 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:04.643987 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:04.643989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:04.644239 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:04.644397 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:04.645065 (+ 668us) server_negotiation.cc:299] Negotiation successful
0807 03:10:04.645283 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":215,"threads_started":1}
I20260807 03:10:04.646649 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:04.628723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:41731 (local address 127.17.31.1:48437)
0807 03:10:04.629402 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:04.629440 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:04.631094 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:04.631715 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:04.631723 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:04.632193 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:10:04.632743 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:04.632760 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:04.633750 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:04.633753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:04.635459 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:04.635473 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:04.636285 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:04.636292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:04.636588 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:04.637310 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:10:04.637336 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:10:04.639770 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:10:04.642839 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:10:04.642848 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:10:04.642862 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:10:04.643256 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:10:04.643661 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:10:04.643666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:10:04.643670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:10:04.643854 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:10:04.644377 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:10:04.644384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:10:04.644697 (+ 313us) client_negotiation.cc:769] Sending connection context
0807 03:10:04.644961 (+ 264us) client_negotiation.cc:240] Negotiation successful
0807 03:10:04.645485 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":272,"threads_started":1}
I20260807 03:10:04.648217 28601 heartbeater.cc:347] Connected to a master server at 127.17.31.62:41731
I20260807 03:10:04.648530 28601 heartbeater.cc:464] Registering TS with master...
I20260807 03:10:04.649436 28601 heartbeater.cc:511] Master 127.17.31.62:41731 requested a full tablet report, sending...
I20260807 03:10:04.651654 28398 ts_manager.cc:194] Registered new tserver with Master: 4df4ec0cae9c428dbb3481243b52d9af (127.17.31.1:42437)
I20260807 03:10:04.653116 28398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:48437
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:10:04.689422 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:41731
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:34329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:10:05.010596 28607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:10:05.010828 28607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:10:05.011000 28607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:10:05.017681 28607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:10:05.017767 28607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:10:05.017886 28607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:10:05.027063 28607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:34329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:41731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:10:05.028214 28607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:10:05.029521 28607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:10:05.042281 28612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:05.042248 28613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:05.042377 28615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:10:05.043200 28607 server_base.cc:1034] running on GCE node
I20260807 03:10:05.043759 28607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:10:05.044646 28607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:10:05.045861 28607 hybrid_clock.cc:648] HybridClock initialized: now 1786072205045820 us; error 55 us; skew 500 ppm
I20260807 03:10:05.046161 28607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:10:05 dist-test-slave-tm1g krb5kdc[28349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072205, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:10:05.049188 28607 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:10:05.050655 28607 webserver.cc:460] Webserver started at http://127.17.31.2:36967/ using document root <none> and password file <none>
I20260807 03:10:05.051275 28607 fs_manager.cc:359] Metadata directory not provided
I20260807 03:10:05.051370 28607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:10:05.051604 28607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:10:05.054208 28607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "c28100dc85d24bdfbe7fb1bd5540e50f"
format_stamp: "Formatted at 2026-08-07 03:10:05 on dist-test-slave-tm1g"
server_key: "ffc93d5913b683c5a2155ecf2a02aabc"
server_key_iv: "a9aafedf76447a6bed07102f1cb67ffb"
server_key_version: "encryptionkey@0"
I20260807 03:10:05.054981 28607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "c28100dc85d24bdfbe7fb1bd5540e50f"
format_stamp: "Formatted at 2026-08-07 03:10:05 on dist-test-slave-tm1g"
server_key: "ffc93d5913b683c5a2155ecf2a02aabc"
server_key_iv: "a9aafedf76447a6bed07102f1cb67ffb"
server_key_version: "encryptionkey@0"
I20260807 03:10:05.059731 28607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260807 03:10:05.062978 28622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:05.064287 28607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:10:05.064456 28607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "c28100dc85d24bdfbe7fb1bd5540e50f"
format_stamp: "Formatted at 2026-08-07 03:10:05 on dist-test-slave-tm1g"
server_key: "ffc93d5913b683c5a2155ecf2a02aabc"
server_key_iv: "a9aafedf76447a6bed07102f1cb67ffb"
server_key_version: "encryptionkey@0"
I20260807 03:10:05.064654 28607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:10:05.079946 28607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:10:05.083544 28607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:10:05.083793 28607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:10:05.084597 28607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:10:05.085762 28607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:10:05.085843 28607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:05.085934 28607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:10:05.085979 28607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:05.118255 28607 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:43191
I20260807 03:10:05.118324 28741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:43191 every 8 connection(s)
I20260807 03:10:05.119504 28607 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:10:05.120868 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 28607
I20260807 03:10:05.121071 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:10:05.121546 17532 external_mini_cluster.cc:1428] Setting key d5e31773399ca9ef883f74e500288096
Aug 07 03:10:05 dist-test-slave-tm1g krb5kdc[28349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072205, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:10:05.140481 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.122055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:36369 (local address 127.17.31.62:41731)
0807 03:10:05.122275 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:05.122282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:05.123429 (+ 1147us) server_negotiation.cc:407] Connection header received
0807 03:10:05.125160 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:05.125166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:05.125248 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:05.125407 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:05.126876 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.127714 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.129257 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.129704 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.129850 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.133393 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:05.133410 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:05.133417 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:10:05.133460 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:05.137095 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:05.137861 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:05.137869 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:05.137875 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:05.138028 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:05.138473 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:05.138480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:05.138486 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:05.138894 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:05.139064 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:05.139768 (+ 704us) server_negotiation.cc:299] Negotiation successful
0807 03:10:05.140023 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 03:10:05.141054 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.122524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:41731 (local address 127.17.31.2:36369)
0807 03:10:05.123266 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:05.123306 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:05.124912 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:05.125624 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:05.125631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:05.126108 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:10:05.126713 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:05.126730 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.127909 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.127914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:05.129111 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:05.129123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.129958 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.129965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:05.130190 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.130915 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:10:05.130937 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:10:05.133178 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:10:05.137300 (+ 4122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:10:05.137307 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:10:05.137317 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:10:05.137721 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:10:05.138132 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:10:05.138135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:10:05.138137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:10:05.138345 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:10:05.139020 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:10:05.139025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:10:05.139300 (+ 275us) client_negotiation.cc:769] Sending connection context
0807 03:10:05.139624 (+ 324us) client_negotiation.cc:240] Negotiation successful
0807 03:10:05.140003 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":324,"threads_started":1}
I20260807 03:10:05.142452 28742 heartbeater.cc:347] Connected to a master server at 127.17.31.62:41731
I20260807 03:10:05.142750 28742 heartbeater.cc:464] Registering TS with master...
I20260807 03:10:05.143441 28742 heartbeater.cc:511] Master 127.17.31.62:41731 requested a full tablet report, sending...
I20260807 03:10:05.144907 28398 ts_manager.cc:194] Registered new tserver with Master: c28100dc85d24bdfbe7fb1bd5540e50f (127.17.31.2:43191)
I20260807 03:10:05.145694 28398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:36369
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:10:05.177201 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:41731
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:34329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:10:05.499655 28748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:10:05.499953 28748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:10:05.500128 28748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:10:05.507550 28748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:10:05.507687 28748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:10:05.507869 28748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:10:05.517305 28748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:34329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:41731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:10:05.518612 28748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:10:05.519903 28748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:10:05.532359 28754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:05.532440 28757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:05.532440 28755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:10:05.533300 28748 server_base.cc:1034] running on GCE node
I20260807 03:10:05.533860 28748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:10:05.534758 28748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:10:05.535946 28748 hybrid_clock.cc:648] HybridClock initialized: now 1786072205535901 us; error 35 us; skew 500 ppm
I20260807 03:10:05.536263 28748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:10:05 dist-test-slave-tm1g krb5kdc[28349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072205, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:10:05.539326 28748 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:10:05.540591 28748 webserver.cc:460] Webserver started at http://127.17.31.3:37629/ using document root <none> and password file <none>
I20260807 03:10:05.541204 28748 fs_manager.cc:359] Metadata directory not provided
I20260807 03:10:05.541318 28748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:10:05.541565 28748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:10:05.544272 28748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "03cbc735d0e94edd98862dd31e4c3ea9"
format_stamp: "Formatted at 2026-08-07 03:10:05 on dist-test-slave-tm1g"
server_key: "c2c66321beef48c319f7051f376dc50b"
server_key_iv: "3ff9ea3698ebe54a0459755ba7167fdc"
server_key_version: "encryptionkey@0"
I20260807 03:10:05.544947 28748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "03cbc735d0e94edd98862dd31e4c3ea9"
format_stamp: "Formatted at 2026-08-07 03:10:05 on dist-test-slave-tm1g"
server_key: "c2c66321beef48c319f7051f376dc50b"
server_key_iv: "3ff9ea3698ebe54a0459755ba7167fdc"
server_key_version: "encryptionkey@0"
I20260807 03:10:05.549727 28748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260807 03:10:05.553004 28764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:05.554243 28748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 03:10:05.554471 28748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "03cbc735d0e94edd98862dd31e4c3ea9"
format_stamp: "Formatted at 2026-08-07 03:10:05 on dist-test-slave-tm1g"
server_key: "c2c66321beef48c319f7051f376dc50b"
server_key_iv: "3ff9ea3698ebe54a0459755ba7167fdc"
server_key_version: "encryptionkey@0"
I20260807 03:10:05.554627 28748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:10:05.573668 28748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:10:05.577221 28748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:10:05.577458 28748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:10:05.578385 28748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:10:05.579581 28748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:10:05.579653 28748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:05.579744 28748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:10:05.579787 28748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:05.612010 28748 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:35853
I20260807 03:10:05.612098 28890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:35853 every 8 connection(s)
I20260807 03:10:05.613219 28748 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:10:05.618763 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 28748
I20260807 03:10:05.618944 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:10:05.619400 17532 external_mini_cluster.cc:1428] Setting key e8ec490b94c562e933dd2f351d47ef21
Aug 07 03:10:05 dist-test-slave-tm1g krb5kdc[28349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072205, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:10:05.632318 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.615589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:46255 (local address 127.17.31.62:41731)
0807 03:10:05.615774 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:05.615779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:05.616804 (+ 1025us) server_negotiation.cc:407] Connection header received
0807 03:10:05.618545 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:05.618554 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:05.618684 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:05.618807 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:05.620474 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.621328 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.622910 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.623314 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.623455 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.626861 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:05.626866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:05.626869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:10:05.626900 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:05.629381 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:05.630088 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:05.630094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:05.630104 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:05.630225 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:05.630665 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:05.630670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:05.630675 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:05.630960 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:05.631116 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:05.631763 (+ 647us) server_negotiation.cc:299] Negotiation successful
0807 03:10:05.631987 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 03:10:05.633309 28893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.615973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:41731 (local address 127.17.31.3:46255)
0807 03:10:05.616640 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:05.616677 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:05.618209 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:05.619102 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:05.619110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:05.619644 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:10:05.620284 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:05.620312 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.621475 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.621481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:05.622755 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:05.622768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.623544 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.623552 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:05.623850 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.624537 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:10:05.624563 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:10:05.626663 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:10:05.629527 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:10:05.629535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:10:05.629548 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:10:05.629958 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:10:05.630358 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:10:05.630364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:10:05.630368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:10:05.630543 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:10:05.631083 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:10:05.631090 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:10:05.631402 (+ 312us) client_negotiation.cc:769] Sending connection context
0807 03:10:05.631659 (+ 257us) client_negotiation.cc:240] Negotiation successful
0807 03:10:05.632136 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":263,"threads_started":1}
I20260807 03:10:05.634748 28891 heartbeater.cc:347] Connected to a master server at 127.17.31.62:41731
I20260807 03:10:05.635049 28891 heartbeater.cc:464] Registering TS with master...
I20260807 03:10:05.635732 28891 heartbeater.cc:511] Master 127.17.31.62:41731 requested a full tablet report, sending...
I20260807 03:10:05.637013 28398 ts_manager.cc:194] Registered new tserver with Master: 03cbc735d0e94edd98862dd31e4c3ea9 (127.17.31.3:35853)
I20260807 03:10:05.637816 28398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:46255
I20260807 03:10:05.646618 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:10:05.656599 28601 heartbeater.cc:503] Master 127.17.31.62:41731 was elected leader, sending a full tablet report...
I20260807 03:10:05.663399 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.650989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.17.31.62:41731)
0807 03:10:05.651138 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:05.651142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:05.651700 (+ 558us) server_negotiation.cc:407] Connection header received
0807 03:10:05.651830 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:05.651834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:05.651890 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:05.651966 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:05.653356 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.654050 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.655278 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.655683 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.655774 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.658279 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:05.658284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:05.658288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:10:05.658368 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:05.661105 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:05.661764 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:05.661768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:05.661770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:05.661900 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:05.662210 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:05.662213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:05.662216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:05.662582 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:05.662756 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:05.663035 (+ 279us) server_negotiation.cc:299] Negotiation successful
0807 03:10:05.663181 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 03:10:05.667263 28398 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:10:05.674207 28398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:10:05.696292 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.685329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:42437 (local address 127.0.0.1:35738)
0807 03:10:05.687181 (+ 1852us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:05.687198 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:05.687333 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:05.688322 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:05.688329 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:05.688359 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:10:05.688783 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:05.688796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.690887 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.690900 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:05.693284 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:05.693302 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.695196 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.695203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:05.695451 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.695551 (+ 100us) client_negotiation.cc:769] Sending connection context
0807 03:10:05.695687 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 03:10:05.695889 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1740,"thread_start_us":351,"threads_started":1}
I20260807 03:10:05.697032 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.685456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.17.31.1:42437)
0807 03:10:05.687001 (+ 1545us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:05.687011 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:05.687299 (+ 288us) server_negotiation.cc:407] Connection header received
0807 03:10:05.687553 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:05.687562 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:05.687829 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:05.687998 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:10:05.689014 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.690680 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.693466 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.694970 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.695118 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.695370 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:05.696580 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0807 03:10:05.696744 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1332,"thread_start_us":1169,"threads_started":1}
I20260807 03:10:05.697533 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.686647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:35853 (local address 127.0.0.1:35768)
0807 03:10:05.689302 (+ 2655us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:05.689326 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:05.689444 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:05.690366 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:05.690371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:05.690400 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:10:05.690991 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:05.691009 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.693106 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.693112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:05.694555 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:05.694567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.696665 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.696671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:05.696859 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.696904 (+ 45us) client_negotiation.cc:769] Sending connection context
0807 03:10:05.697001 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 03:10:05.697129 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2516,"mutex_wait_us":4,"thread_start_us":668,"threads_started":1}
I20260807 03:10:05.697847 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.687164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35768 (local address 127.17.31.3:35853)
0807 03:10:05.687828 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:05.687835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:05.689545 (+ 1710us) server_negotiation.cc:407] Connection header received
0807 03:10:05.689739 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:05.689746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:05.689936 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:05.690197 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:10:05.691318 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.692924 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.694704 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.696196 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.696314 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.696534 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:05.697367 (+ 833us) server_negotiation.cc:299] Negotiation successful
0807 03:10:05.697539 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":314,"threads_started":1}
I20260807 03:10:05.699654 28906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.685285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:43191 (local address 127.0.0.1:33556)
0807 03:10:05.686057 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:05.686098 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:05.686278 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:05.690272 (+ 3994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:05.690279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:05.690326 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:10:05.690831 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:05.690846 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.694775 (+ 3929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.694782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:05.696904 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:05.696914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.698745 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.698752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:05.698925 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.698963 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 03:10:05.699200 (+ 237us) client_negotiation.cc:240] Negotiation successful
0807 03:10:05.699334 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":3695,"threads_started":1}
I20260807 03:10:05.699754 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.685873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33556 (local address 127.17.31.2:43191)
0807 03:10:05.688106 (+ 2233us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:05.688114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:05.688143 (+ 29us) server_negotiation.cc:407] Connection header received
0807 03:10:05.688245 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:05.688252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:05.688488 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:05.688696 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:10:05.691822 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.694583 (+ 2761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.697060 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.698518 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.698653 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.698821 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:05.699252 (+ 431us) server_negotiation.cc:299] Negotiation successful
0807 03:10:05.699438 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2073,"thread_start_us":1930,"threads_started":1}
I20260807 03:10:05.701262 28817 tablet_service.cc:1467] Processing CreateTablet for tablet a354ffc8553b4a5fb62980887d4d4a17 (DEFAULT_TABLE table=test-table [id=23800144f7ca4aa8801ecbbafeab2c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:10:05.701260 28533 tablet_service.cc:1467] Processing CreateTablet for tablet a354ffc8553b4a5fb62980887d4d4a17 (DEFAULT_TABLE table=test-table [id=23800144f7ca4aa8801ecbbafeab2c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:10:05.702780 28533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a354ffc8553b4a5fb62980887d4d4a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:10:05.702780 28817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a354ffc8553b4a5fb62980887d4d4a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:10:05.703168 28675 tablet_service.cc:1467] Processing CreateTablet for tablet a354ffc8553b4a5fb62980887d4d4a17 (DEFAULT_TABLE table=test-table [id=23800144f7ca4aa8801ecbbafeab2c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:10:05.704205 28675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a354ffc8553b4a5fb62980887d4d4a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:10:05.714975 28911 tablet_bootstrap.cc:492] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f: Bootstrap starting.
I20260807 03:10:05.717227 28912 tablet_bootstrap.cc:492] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af: Bootstrap starting.
I20260807 03:10:05.718968 28913 tablet_bootstrap.cc:492] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9: Bootstrap starting.
I20260807 03:10:05.720368 28911 tablet_bootstrap.cc:654] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f: Neither blocks nor log segments found. Creating new log.
I20260807 03:10:05.720718 28912 tablet_bootstrap.cc:654] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af: Neither blocks nor log segments found. Creating new log.
I20260807 03:10:05.721674 28911 log.cc:824] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f: Log is configured to *not* fsync() on all Append() calls
I20260807 03:10:05.721730 28913 tablet_bootstrap.cc:654] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9: Neither blocks nor log segments found. Creating new log.
I20260807 03:10:05.721848 28912 log.cc:824] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af: Log is configured to *not* fsync() on all Append() calls
I20260807 03:10:05.722990 28913 log.cc:824] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9: Log is configured to *not* fsync() on all Append() calls
I20260807 03:10:05.724807 28911 tablet_bootstrap.cc:492] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f: No bootstrap required, opened a new log
I20260807 03:10:05.724911 28912 tablet_bootstrap.cc:492] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af: No bootstrap required, opened a new log
I20260807 03:10:05.725108 28911 ts_tablet_manager.cc:1397] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260807 03:10:05.725174 28912 ts_tablet_manager.cc:1397] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260807 03:10:05.725912 28913 tablet_bootstrap.cc:492] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9: No bootstrap required, opened a new log
I20260807 03:10:05.726147 28913 ts_tablet_manager.cc:1397] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260807 03:10:05.732391 28911 raft_consensus.cc:348] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4ec0cae9c428dbb3481243b52d9af" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42437 } } peers { permanent_uuid: "c28100dc85d24bdfbe7fb1bd5540e50f" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43191 } } peers { permanent_uuid: "03cbc735d0e94edd98862dd31e4c3ea9" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35853 } }
I20260807 03:10:05.732741 28911 raft_consensus.cc:374] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:10:05.732861 28911 raft_consensus.cc:729] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c28100dc85d24bdfbe7fb1bd5540e50f, State: Initialized, Role: FOLLOWER
I20260807 03:10:05.733462 28911 consensus_queue.cc:260] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4ec0cae9c428dbb3481243b52d9af" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42437 } } peers { permanent_uuid: "c28100dc85d24bdfbe7fb1bd5540e50f" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43191 } } peers { permanent_uuid: "03cbc735d0e94edd98862dd31e4c3ea9" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35853 } }
I20260807 03:10:05.734567 28742 heartbeater.cc:503] Master 127.17.31.62:41731 was elected leader, sending a full tablet report...
I20260807 03:10:05.735167 28911 ts_tablet_manager.cc:1428] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260807 03:10:05.735765 28912 raft_consensus.cc:348] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4ec0cae9c428dbb3481243b52d9af" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42437 } } peers { permanent_uuid: "c28100dc85d24bdfbe7fb1bd5540e50f" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43191 } } peers { permanent_uuid: "03cbc735d0e94edd98862dd31e4c3ea9" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35853 } }
I20260807 03:10:05.736196 28912 raft_consensus.cc:374] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:10:05.736290 28912 raft_consensus.cc:729] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df4ec0cae9c428dbb3481243b52d9af, State: Initialized, Role: FOLLOWER
I20260807 03:10:05.736864 28912 consensus_queue.cc:260] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4ec0cae9c428dbb3481243b52d9af" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42437 } } peers { permanent_uuid: "c28100dc85d24bdfbe7fb1bd5540e50f" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43191 } } peers { permanent_uuid: "03cbc735d0e94edd98862dd31e4c3ea9" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35853 } }
I20260807 03:10:05.737092 28913 raft_consensus.cc:348] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4ec0cae9c428dbb3481243b52d9af" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42437 } } peers { permanent_uuid: "c28100dc85d24bdfbe7fb1bd5540e50f" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43191 } } peers { permanent_uuid: "03cbc735d0e94edd98862dd31e4c3ea9" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35853 } }
I20260807 03:10:05.737560 28913 raft_consensus.cc:374] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:10:05.737677 28913 raft_consensus.cc:729] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03cbc735d0e94edd98862dd31e4c3ea9, State: Initialized, Role: FOLLOWER
I20260807 03:10:05.738384 28913 consensus_queue.cc:260] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4ec0cae9c428dbb3481243b52d9af" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42437 } } peers { permanent_uuid: "c28100dc85d24bdfbe7fb1bd5540e50f" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43191 } } peers { permanent_uuid: "03cbc735d0e94edd98862dd31e4c3ea9" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35853 } }
I20260807 03:10:05.742040 28912 ts_tablet_manager.cc:1428] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260807 03:10:05.743289 28891 heartbeater.cc:503] Master 127.17.31.62:41731 was elected leader, sending a full tablet report...
I20260807 03:10:05.743726 28913 ts_tablet_manager.cc:1428] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9: Time spent starting tablet: real 0.017s user 0.018s sys 0.001s
I20260807 03:10:05.814065 28917 raft_consensus.cc:482] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:10:05.814393 28917 raft_consensus.cc:504] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4ec0cae9c428dbb3481243b52d9af" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42437 } } peers { permanent_uuid: "c28100dc85d24bdfbe7fb1bd5540e50f" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43191 } } peers { permanent_uuid: "03cbc735d0e94edd98862dd31e4c3ea9" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35853 } }
I20260807 03:10:05.816152 28917 leader_election.cc:288] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4df4ec0cae9c428dbb3481243b52d9af (127.17.31.1:42437), 03cbc735d0e94edd98862dd31e4c3ea9 (127.17.31.3:35853)
I20260807 03:10:05.823091 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.816716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:40669 (local address 127.17.31.1:42437)
0807 03:10:05.816948 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:05.816957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:05.817117 (+ 160us) server_negotiation.cc:407] Connection header received
0807 03:10:05.817350 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:05.817357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:05.817451 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:05.817545 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:10:05.818179 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.819180 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.821169 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.822069 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.822167 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.822237 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:05.822694 (+ 457us) server_negotiation.cc:299] Negotiation successful
0807 03:10:05.822827 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 03:10:05.823134 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.816349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:42437 (local address 127.17.31.2:40669)
0807 03:10:05.817001 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:05.817018 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:05.817192 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:05.817621 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:05.817624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:05.817646 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:10:05.818004 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:05.818015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.819308 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.819313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:05.821008 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:05.821022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.822283 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.822289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:05.822489 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.822538 (+ 49us) client_negotiation.cc:769] Sending connection context
0807 03:10:05.822624 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 03:10:05.822763 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":26,"thread_start_us":326,"threads_started":1}
I20260807 03:10:05.823539 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.817302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:43205 (local address 127.17.31.3:35853)
0807 03:10:05.817509 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:05.817526 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:05.817858 (+ 332us) server_negotiation.cc:407] Connection header received
0807 03:10:05.817970 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:05.817977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:05.818068 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:05.818154 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:10:05.818798 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.819713 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.821582 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.822482 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.822622 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.822718 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:05.823191 (+ 473us) server_negotiation.cc:299] Negotiation successful
0807 03:10:05.823309 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 03:10:05.823695 28922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.817149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:35853 (local address 127.17.31.2:43205)
0807 03:10:05.817746 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:05.817762 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:05.817868 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:05.818184 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:05.818188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:05.818210 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:10:05.818601 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:05.818612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.819871 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.819874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:05.821429 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:05.821443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.822768 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.822774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:05.822998 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.823027 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 03:10:05.823147 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 03:10:05.823299 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":23,"thread_start_us":275,"threads_started":1}
I20260807 03:10:05.824508 28554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a354ffc8553b4a5fb62980887d4d4a17" candidate_uuid: "c28100dc85d24bdfbe7fb1bd5540e50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df4ec0cae9c428dbb3481243b52d9af" is_pre_election: true
I20260807 03:10:05.824676 28839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a354ffc8553b4a5fb62980887d4d4a17" candidate_uuid: "c28100dc85d24bdfbe7fb1bd5540e50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03cbc735d0e94edd98862dd31e4c3ea9" is_pre_election: true
I20260807 03:10:05.825057 28554 raft_consensus.cc:2454] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c28100dc85d24bdfbe7fb1bd5540e50f in term 0.
I20260807 03:10:05.825189 28839 raft_consensus.cc:2454] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c28100dc85d24bdfbe7fb1bd5540e50f in term 0.
I20260807 03:10:05.825814 28623 leader_election.cc:302] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df4ec0cae9c428dbb3481243b52d9af, c28100dc85d24bdfbe7fb1bd5540e50f; no voters:
I20260807 03:10:05.826177 28917 raft_consensus.cc:2781] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:10:05.826340 28917 raft_consensus.cc:482] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:10:05.826476 28917 raft_consensus.cc:3037] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:10:05.828681 28917 raft_consensus.cc:504] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4ec0cae9c428dbb3481243b52d9af" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42437 } } peers { permanent_uuid: "c28100dc85d24bdfbe7fb1bd5540e50f" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43191 } } peers { permanent_uuid: "03cbc735d0e94edd98862dd31e4c3ea9" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35853 } }
I20260807 03:10:05.829236 28917 leader_election.cc:288] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [CANDIDATE]: Term 1 election: Requested vote from peers 4df4ec0cae9c428dbb3481243b52d9af (127.17.31.1:42437), 03cbc735d0e94edd98862dd31e4c3ea9 (127.17.31.3:35853)
I20260807 03:10:05.829631 28554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a354ffc8553b4a5fb62980887d4d4a17" candidate_uuid: "c28100dc85d24bdfbe7fb1bd5540e50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4df4ec0cae9c428dbb3481243b52d9af"
I20260807 03:10:05.829715 28839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a354ffc8553b4a5fb62980887d4d4a17" candidate_uuid: "c28100dc85d24bdfbe7fb1bd5540e50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03cbc735d0e94edd98862dd31e4c3ea9"
I20260807 03:10:05.829820 28554 raft_consensus.cc:3037] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:10:05.829952 28839 raft_consensus.cc:3037] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:10:05.831992 28554 raft_consensus.cc:2454] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c28100dc85d24bdfbe7fb1bd5540e50f in term 1.
I20260807 03:10:05.832218 28839 raft_consensus.cc:2454] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c28100dc85d24bdfbe7fb1bd5540e50f in term 1.
I20260807 03:10:05.832394 28623 leader_election.cc:302] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4df4ec0cae9c428dbb3481243b52d9af, c28100dc85d24bdfbe7fb1bd5540e50f; no voters:
I20260807 03:10:05.832679 28917 raft_consensus.cc:2781] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:10:05.833107 28917 raft_consensus.cc:686] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [term 1 LEADER]: Becoming Leader. State: Replica: c28100dc85d24bdfbe7fb1bd5540e50f, State: Running, Role: LEADER
I20260807 03:10:05.833567 28917 consensus_queue.cc:237] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4ec0cae9c428dbb3481243b52d9af" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42437 } } peers { permanent_uuid: "c28100dc85d24bdfbe7fb1bd5540e50f" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43191 } } peers { permanent_uuid: "03cbc735d0e94edd98862dd31e4c3ea9" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35853 } }
I20260807 03:10:05.838145 28398 catalog_manager.cc:5495] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f reported cstate change: term changed from 0 to 1, leader changed from <none> to c28100dc85d24bdfbe7fb1bd5540e50f (127.17.31.2). New cstate: current_term: 1 leader_uuid: "c28100dc85d24bdfbe7fb1bd5540e50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4df4ec0cae9c428dbb3481243b52d9af" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c28100dc85d24bdfbe7fb1bd5540e50f" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 43191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03cbc735d0e94edd98862dd31e4c3ea9" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35853 } health_report { overall_health: UNKNOWN } } }
W20260807 03:10:05.864784 28892 tablet.cc:2369] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:10:05.871371 28743 tablet.cc:2369] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:10:05.878021 28602 tablet.cc:2369] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:10:05.888697 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.883939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.17.31.2:43191)
0807 03:10:05.884110 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:05.884114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:05.884130 (+ 16us) server_negotiation.cc:407] Connection header received
0807 03:10:05.884283 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:05.884287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:05.884365 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:05.884478 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:10:05.884792 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.885532 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.886843 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.887215 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.887305 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.887674 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:10:05.888141 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:10:05.888254 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:05.888332 (+ 78us) server_negotiation.cc:299] Negotiation successful
0807 03:10:05.888391 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 03:10:05.896382 28839 raft_consensus.cc:1261] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9 [term 1 FOLLOWER]: Refusing update from remote peer c28100dc85d24bdfbe7fb1bd5540e50f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:10:05.896385 28554 raft_consensus.cc:1261] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af [term 1 FOLLOWER]: Refusing update from remote peer c28100dc85d24bdfbe7fb1bd5540e50f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:10:05.897270 28923 consensus_queue.cc:1035] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "03cbc735d0e94edd98862dd31e4c3ea9" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:10:05.897615 28917 consensus_queue.cc:1035] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4df4ec0cae9c428dbb3481243b52d9af" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:10:05.909060 28926 mvcc.cc:205] Tried to move back new op lower bound from 7315751755342254080 to 7315751755097903104. Current Snapshot: MvccSnapshot[applied={T|T < 7315751755342254080}]
I20260807 03:10:05.910694 28927 mvcc.cc:205] Tried to move back new op lower bound from 7315751755342254080 to 7315751755097903104. Current Snapshot: MvccSnapshot[applied={T|T < 7315751755342254080}]
I20260807 03:10:05.920989 28398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:10:05.921236 28398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:10:05.924315 28398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4390780bf82e45c0896fdeffebd2354e: Sending DeleteTablet for 3 replicas of tablet a354ffc8553b4a5fb62980887d4d4a17
I20260807 03:10:05.925406 28533 tablet_service.cc:1514] Processing DeleteTablet for tablet a354ffc8553b4a5fb62980887d4d4a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:10:05 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:35738
I20260807 03:10:05.925540 28675 tablet_service.cc:1514] Processing DeleteTablet for tablet a354ffc8553b4a5fb62980887d4d4a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:10:05 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:33556
I20260807 03:10:05.926270 28936 tablet_replica.cc:330] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af: stopping tablet replica
I20260807 03:10:05.926476 28817 tablet_service.cc:1514] Processing DeleteTablet for tablet a354ffc8553b4a5fb62980887d4d4a17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:10:05 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:35768
I20260807 03:10:05.926743 28936 raft_consensus.cc:2229] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:10:05.927217 28936 raft_consensus.cc:2258] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:10:05.928632 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 28464
I20260807 03:10:05.929283 28936 ts_tablet_manager.cc:1905] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:10:05.937389 28936 ts_tablet_manager.cc:1918] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:10:05.937602 28936 log.cc:1196] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/a354ffc8553b4a5fb62980887d4d4a17
I20260807 03:10:05.938231 28936 ts_tablet_manager.cc:1939] T a354ffc8553b4a5fb62980887d4d4a17 P 4df4ec0cae9c428dbb3481243b52d9af: Deleting consensus metadata
I20260807 03:10:05.940158 28382 catalog_manager.cc:4841] TS 4df4ec0cae9c428dbb3481243b52d9af (127.17.31.1:42437): tablet a354ffc8553b4a5fb62980887d4d4a17 (table test-table [id=23800144f7ca4aa8801ecbbafeab2c9d]) successfully deleted
I20260807 03:10:05.951524 28939 tablet_replica.cc:330] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9: stopping tablet replica
I20260807 03:10:05.952168 28939 raft_consensus.cc:2229] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:10:05.952759 28939 raft_consensus.cc:2258] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:10:05.954967 28939 ts_tablet_manager.cc:1905] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:10:05.958745 28938 tablet_replica.cc:330] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f: stopping tablet replica
I20260807 03:10:05.959232 28938 raft_consensus.cc:2229] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:10:05.960024 28938 raft_consensus.cc:2258] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 07 03:10:05 dist-test-slave-tm1g krb5kdc[28349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:10:05.962167 28938 ts_tablet_manager.cc:1905] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:10:05.962448 28939 ts_tablet_manager.cc:1918] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:10:05.962633 28939 log.cc:1196] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/a354ffc8553b4a5fb62980887d4d4a17
I20260807 03:10:05.963171 28939 ts_tablet_manager.cc:1939] T a354ffc8553b4a5fb62980887d4d4a17 P 03cbc735d0e94edd98862dd31e4c3ea9: Deleting consensus metadata
I20260807 03:10:05.964948 28382 catalog_manager.cc:4841] TS 03cbc735d0e94edd98862dd31e4c3ea9 (127.17.31.3:35853): tablet a354ffc8553b4a5fb62980887d4d4a17 (table test-table [id=23800144f7ca4aa8801ecbbafeab2c9d]) successfully deleted
I20260807 03:10:05.970073 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:05.930586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35742 (local address 127.17.31.1:42437)
0807 03:10:05.930749 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:05.930755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:05.950864 (+ 20109us) server_negotiation.cc:407] Connection header received
0807 03:10:05.951086 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:05.951093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:05.951187 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:05.951285 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:05.953408 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.954581 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.956419 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:05.957057 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:05.957145 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:05.962696 (+ 5551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:05.962710 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:05.962727 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:10:05.962768 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:05.966632 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:05.967542 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:05.967551 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:05.967554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:05.967712 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:05.968193 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:05.968198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:05.968202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:05.968647 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:05.968954 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:05.969315 (+ 361us) server_negotiation.cc:299] Negotiation successful
0807 03:10:05.969537 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 03:10:05.970432 28938 ts_tablet_manager.cc:1918] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:10:05.970662 28938 log.cc:1196] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/a354ffc8553b4a5fb62980887d4d4a17
I20260807 03:10:05.970772 28596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:10:05.971246 28938 ts_tablet_manager.cc:1939] T a354ffc8553b4a5fb62980887d4d4a17 P c28100dc85d24bdfbe7fb1bd5540e50f: Deleting consensus metadata
I20260807 03:10:05.973313 28383 catalog_manager.cc:4841] TS c28100dc85d24bdfbe7fb1bd5540e50f (127.17.31.2:43191): tablet a354ffc8553b4a5fb62980887d4d4a17 (table test-table [id=23800144f7ca4aa8801ecbbafeab2c9d]) successfully deleted
I20260807 03:10:06.082075 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 28464
I20260807 03:10:06.101806 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 28607
Aug 07 03:10:06 dist-test-slave-tm1g krb5kdc[28349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:10:06.120095 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:06.102630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33570 (local address 127.17.31.2:43191)
0807 03:10:06.102804 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:06.102808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:06.102890 (+ 82us) server_negotiation.cc:407] Connection header received
0807 03:10:06.103085 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:06.103089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:06.103160 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:06.103240 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:06.104855 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:06.105692 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:06.107579 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:06.107999 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:06.108093 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:06.113083 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:06.113095 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:06.113109 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:10:06.113150 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:06.116975 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:06.117715 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:06.117723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:06.117727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:06.117899 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:06.118291 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:06.118296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:06.118326 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:06.118822 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:06.119134 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:06.119443 (+ 309us) server_negotiation.cc:299] Negotiation successful
0807 03:10:06.119661 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:10:06.128510 28737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:10:06.240193 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 28607
I20260807 03:10:06.260697 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 28748
Aug 07 03:10:06 dist-test-slave-tm1g krb5kdc[28349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:10:06.275691 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:06.261443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35772 (local address 127.17.31.3:35853)
0807 03:10:06.261634 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:06.261640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:06.261738 (+ 98us) server_negotiation.cc:407] Connection header received
0807 03:10:06.261955 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:06.261961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:06.262037 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:06.262139 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:06.263469 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:06.264284 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:06.265522 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:06.265916 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:06.266067 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:06.270382 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:06.270396 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:06.270412 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:10:06.270446 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:06.272920 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:06.273567 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:06.273575 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:06.273577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:06.273697 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:06.274054 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:06.274060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:06.274063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:06.274566 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:06.274880 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:06.275160 (+ 280us) server_negotiation.cc:299] Negotiation successful
0807 03:10:06.275368 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 03:10:06.276295 28886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:10:06.362439 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 28748
I20260807 03:10:06.380023 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 28366
I20260807 03:10:06.381099 28431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:10:06.488082 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 28366
2026-08-07T03:10:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4231 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:10:06 dist-test-slave-tm1g krb5kdc[28948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:10:06 dist-test-slave-tm1g krb5kdc[28948](info): set up 2 sockets
Aug 07 03:10:06 dist-test-slave-tm1g krb5kdc[28948](info): commencing operation
krb5kdc: starting...
W20260807 03:10:07.608577 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:10:07 dist-test-slave-tm1g krb5kdc[28948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:10:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:10:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.17.31.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.17.31.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.17.31.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/rangeradmin_127.17.31.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.17.31.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/rangeradmin_127.17.31.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.17.31.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.17.31.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.17.31.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/rangerlookup_127.17.31.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.17.31.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/rangerlookup_127.17.31.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.31.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.17.31.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.31.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/HTTP_127.17.31.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.31.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/HTTP_127.17.31.20@KRBTEST.COM.keytab.
I20260807 03:10:07.943110 17532 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasko9xIn4/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/postgres -l logfile start
2026-08-07 03:10:10.329 UTC [28978] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 03:10:10.329 UTC [28978] LOG: listening on IPv4 address "127.17.31.20", port 54433
2026-08-07 03:10:10.332 UTC [28978] LOG: listening on Unix socket "/tmp/.s.PGSQL.54433"
2026-08-07 03:10:10.354 UTC [28982] LOG: database system was shut down at 2026-08-07 03:10:08 UTC
2026-08-07 03:10:10.358 UTC [28978] LOG: database system is ready to accept connections
I20260807 03:10:11.348055 17532 mini_postgres.cc:96] Postgres bound to 54433
2026-08-07 03:10:11.369 UTC [28991] FATAL: database "slave" does not exist
127.17.31.20:54433 - accepting connections
I20260807 03:10:11.369831 17532 mini_ranger.cc:161] Starting Ranger...
I20260807 03:10:11.395984 17532 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 03:10:11.810115 17532 mini_ranger.cc:87] Created ranger Postgres database
I20260807 03:10:11.810294 17532 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-admin
2026-08-07 03:10:12,098 [I] DB FLAVOR :POSTGRES
2026-08-07 03:10:12,099 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 03:10:12,099 [I] Checking connection..
2026-08-07 03:10:12,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 03:10:12,346 [I] Checking connection passed.
2026-08-07 03:10:12,346 [I] --------- Verifying version history table ---------
2026-08-07 03:10:12,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 03:10:12,577 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 03:10:12,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 03:10:12,834 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 03:10:12,835 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 03:10:12,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 03:10:13.086 UTC [29059] WARNING: there is no transaction in progress
2026-08-07 03:10:13,093 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 03:10:13,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 03:10:13,350 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 03:10:13,350 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 03:10:13,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 03:10:13,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 03:10:13,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tm1g.c.gcp-upstream.internal','N') ;"
2026-08-07 03:10:14,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 03:10:14,312 [I] Table x_portal_user does not exist in database ranger
2026-08-07 03:10:14,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 03:10:14,551 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 03:10:14,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 03:10:14,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 03:10:15,052 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 03:10:15,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 03:10:15.969 UTC [29171] WARNING: there is no transaction in progress
2026-08-07 03:10:15.985 UTC [29171] WARNING: there is no transaction in progress
2026-08-07 03:10:16.000 UTC [29171] WARNING: there is no transaction in progress
2026-08-07 03:10:16.015 UTC [29171] WARNING: there is no transaction in progress
2026-08-07 03:10:16.032 UTC [29171] WARNING: there is no transaction in progress
2026-08-07 03:10:16.052 UTC [29171] WARNING: there is no transaction in progress
2026-08-07 03:10:16.114 UTC [29171] WARNING: there is no transaction in progress
2026-08-07 03:10:16.120 UTC [29171] WARNING: there is no transaction in progress
2026-08-07 03:10:16.133 UTC [29171] WARNING: there is no transaction in progress
2026-08-07 03:10:16.143 UTC [29171] WARNING: there is no transaction in progress
2026-08-07 03:10:16.152 UTC [29171] WARNING: there is no transaction in progress
2026-08-07 03:10:16.163 UTC [29171] WARNING: there is no transaction in progress
2026-08-07 03:10:16.172 UTC [29171] WARNING: there is no transaction in progress
2026-08-07 03:10:16.178 UTC [29171] WARNING: there is no transaction in progress
2026-08-07 03:10:16.184 UTC [29171] WARNING: there is no transaction in progress
2026-08-07 03:10:16.190 UTC [29171] WARNING: there is no transaction in progress
2026-08-07 03:10:16,753 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 03:10:16,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 03:10:16,989 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 03:10:16,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 03:10:17,240 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 03:10:17,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 03:10:17,481 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 03:10:17,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 03:10:17,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 03:10:17,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tm1g.c.gcp-upstream.internal';"
2026-08-07 03:10:18,230 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 03:10:18,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 03:10:18,477 [I] DB_PATCHES have already been applied
I20260807 03:10:18.483814 17532 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-admin:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasko9xIn4/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 03:10:18.483916 17532 mini_ranger.cc:193] Using host: 127.17.31.20
I20260807 03:10:18.502044 17532 mini_ranger.cc:227] Ranger admin URL: http://127.17.31.20:56457
Aug 07, 2026 3:10:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 3:10:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 3:10:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 3:10:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 3:10:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 3:10:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 3:10:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 03:10:19.656719 17532 mini_ranger.cc:160] Time spent starting Ranger: real 8.287s user 0.007s sys 0.039s
Aug 07, 2026 3:10:39 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 3:10:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 07, 2026 3:10:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 3:10:39 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 3:10:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 3:10:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 3:10:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 3:10:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 3:10:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 3:10:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 3:10:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 3:10:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 3:10:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 3:10:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 3:10:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 3:10:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 3:10:39 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 3:10:40 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 03:10:40.706036 17532 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.17.31.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.17.31.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.17.31.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/rangerkms_127.17.31.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.17.31.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/rangerkms_127.17.31.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.31.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.17.31.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.31.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/HTTP_127.17.31.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.31.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/HTTP_127.17.31.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 03:10:40 dist-test-slave-tm1g krb5kdc[28948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072240, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 03:10:40.890470 17532 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 03:10:40.913020 17532 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 03:10:41.680565 17532 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 03:10:41.680709 17532 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-kms
I20260807 03:10:41.680860 17532 mini_ranger_kms.cc:227] Using postgres at 127.17.31.20:54433
2026-08-07 03:10:42,167 [I] DB FLAVOR :POSTGRES
2026-08-07 03:10:42,168 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 03:10:42,168 [I] Checking connection
2026-08-07 03:10:42,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 03:10:42,407 [I] connection success
2026-08-07 03:10:42,407 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 03:10:42,407 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 03:10:42,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 03:10:42,642 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 03:10:42,642 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 03:10:42,642 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 03:10:42,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:54433/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 03:10:42,890 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 03:10:43.325484 17532 mini_ranger_kms.cc:310] Created kms service
I20260807 03:10:43.501576 17532 mini_ranger_kms.cc:326] Created kudu user
I20260807 03:10:43.546522 17532 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 03:10:43 dist-test-slave-tm1g krb5kdc[28948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072240, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
I20260807 03:10:43.954882 17532 mini_ranger_kms.cc:384] Added ranger policy
I20260807 03:10:43.956336 17532 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-kms:/tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 03:10:43.956420 17532 mini_ranger_kms.cc:240] Using host: 127.17.31.20
I20260807 03:10:43.980499 17532 mini_ranger_kms.cc:276] Ranger KMS PID: 29360
I20260807 03:10:43.980683 17532 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.17.31.20:54919
Aug 07, 2026 3:10:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 3:10:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 3:10:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 3:10:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 3:10:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 3:10:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 03:10:46 dist-test-slave-tm1g krb5kdc[28948](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786072246, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.31.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:10:46.300392 17532 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.410s user 0.136s sys 0.380s
I20260807 03:10:46.300729 17532 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 03:10:46.300930 17532 mini_ranger_kms.cc:401] 127.17.31.20:54919/kms/v1/keys
Aug 07 03:10:46 dist-test-slave-tm1g krb5kdc[28948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072246, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.31.20@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
Aug 07 03:10:47 dist-test-slave-tm1g krb5kdc[28948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072246, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.31.20@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
Aug 07 03:10:49 dist-test-slave-tm1g krb5kdc[28948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:10:49.420112 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:44213
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:44213
--ranger_config_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.31.20:54919/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:10:49.934237 29414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:10:49.934636 29414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:10:49.934780 29414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:10:49.942566 29414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:10:49.942736 29414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:10:49.942816 29414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:10:49.942924 29414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:10:49.943011 29414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:10:49.954375 29414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/wal
--ranger_kms_url=127.17.31.20:54919/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:44213
--ranger_config_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:44213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:10:49.955610 29414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:10:49.956892 29414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:10:49.972342 29422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:49.972915 29419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:49.973166 29420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:10:49.974273 29414 server_base.cc:1034] running on GCE node
I20260807 03:10:49.975144 29414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:10:49.976598 29414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:10:49.977886 29414 hybrid_clock.cc:648] HybridClock initialized: now 1786072249977795 us; error 37 us; skew 500 ppm
I20260807 03:10:49.978392 29414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:10:49 dist-test-slave-tm1g krb5kdc[28948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072249, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:10:49.982497 29414 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:10:49.984117 29414 webserver.cc:460] Webserver started at http://127.17.31.62:34791/ using document root <none> and password file <none>
I20260807 03:10:49.984828 29414 fs_manager.cc:359] Metadata directory not provided
I20260807 03:10:49.984987 29414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:10:49.985284 29414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 03:10:49 dist-test-slave-tm1g krb5kdc[28948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072249, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
I20260807 03:10:50.035588 29414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "e17ba01511704ddcbd0e010a716821cd"
format_stamp: "Formatted at 2026-08-07 03:10:50 on dist-test-slave-tm1g"
server_key: "8db1501861009ef3a12095c1f31d7bdb"
server_key_iv: "5e9c6e7f4eb5d38d114489216b7ed994"
server_key_version: "kuduclusterkey@0"
I20260807 03:10:50.036782 29414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "e17ba01511704ddcbd0e010a716821cd"
format_stamp: "Formatted at 2026-08-07 03:10:50 on dist-test-slave-tm1g"
server_key: "8db1501861009ef3a12095c1f31d7bdb"
server_key_iv: "5e9c6e7f4eb5d38d114489216b7ed994"
server_key_version: "kuduclusterkey@0"
I20260807 03:10:50.043820 29414 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260807 03:10:50.065099 29429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:50.066784 29414 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 03:10:50.067060 29414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "e17ba01511704ddcbd0e010a716821cd"
format_stamp: "Formatted at 2026-08-07 03:10:50 on dist-test-slave-tm1g"
server_key: "8db1501861009ef3a12095c1f31d7bdb"
server_key_iv: "5e9c6e7f4eb5d38d114489216b7ed994"
server_key_version: "kuduclusterkey@0"
I20260807 03:10:50.067329 29414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:10:50.098888 29414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:10:50.103962 29414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:10:50.104393 29414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:10:50.145370 29414 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:44213
I20260807 03:10:50.145424 29484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:44213 every 8 connection(s)
I20260807 03:10:50.146845 29414 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:10:50.150275 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 29414
I20260807 03:10:50.150559 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:10:50.151537 17532 external_mini_cluster.cc:1428] Setting key a79b7a324b2ab4d98b0abfebd93751f1
I20260807 03:10:50.151840 29485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:10:50.173717 29485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd: Bootstrap starting.
I20260807 03:10:50.178591 29485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd: Neither blocks nor log segments found. Creating new log.
I20260807 03:10:50.180533 29485 log.cc:824] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd: Log is configured to *not* fsync() on all Append() calls
I20260807 03:10:50.184290 29485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd: No bootstrap required, opened a new log
Aug 07 03:10:50 dist-test-slave-tm1g krb5kdc[28948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:10:50.195341 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:50.154124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.17.31.62:44213)
0807 03:10:50.170890 (+ 16766us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:50.170907 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:50.170982 (+ 75us) server_negotiation.cc:407] Connection header received
0807 03:10:50.174675 (+ 3693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:50.174703 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:50.175282 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:50.175742 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:50.177375 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:50.179947 (+ 2572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:50.181786 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:50.182492 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:50.182686 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:50.186828 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:50.186839 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:50.186854 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:10:50.186892 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:50.191003 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:50.191704 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:50.191710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:50.191716 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:50.191957 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:50.192352 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:50.192357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:50.192360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:50.192972 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:50.193332 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:50.193804 (+ 472us) server_negotiation.cc:299] Negotiation successful
0807 03:10:50.194224 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16490,"thread_start_us":14618,"threads_started":1}
I20260807 03:10:50.200054 29485 raft_consensus.cc:348] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e17ba01511704ddcbd0e010a716821cd" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44213 } }
I20260807 03:10:50.200641 29485 raft_consensus.cc:374] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:10:50.200851 29485 raft_consensus.cc:729] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e17ba01511704ddcbd0e010a716821cd, State: Initialized, Role: FOLLOWER
I20260807 03:10:50.201632 29485 consensus_queue.cc:260] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e17ba01511704ddcbd0e010a716821cd" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44213 } }
I20260807 03:10:50.201992 29485 raft_consensus.cc:388] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:10:50.202189 29485 raft_consensus.cc:482] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:10:50.202435 29485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:10:50.205974 29485 raft_consensus.cc:504] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e17ba01511704ddcbd0e010a716821cd" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44213 } }
I20260807 03:10:50.206586 29485 leader_election.cc:302] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e17ba01511704ddcbd0e010a716821cd; no voters:
I20260807 03:10:50.207331 29485 leader_election.cc:288] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:10:50.207513 29491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:10:50.208999 29491 raft_consensus.cc:686] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [term 1 LEADER]: Becoming Leader. State: Replica: e17ba01511704ddcbd0e010a716821cd, State: Running, Role: LEADER
I20260807 03:10:50.209831 29485 sys_catalog.cc:564] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:10:50.209864 29491 consensus_queue.cc:237] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e17ba01511704ddcbd0e010a716821cd" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44213 } }
I20260807 03:10:50.217289 29492 sys_catalog.cc:455] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e17ba01511704ddcbd0e010a716821cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e17ba01511704ddcbd0e010a716821cd" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44213 } } }
I20260807 03:10:50.217407 29493 sys_catalog.cc:455] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e17ba01511704ddcbd0e010a716821cd. Latest consensus state: current_term: 1 leader_uuid: "e17ba01511704ddcbd0e010a716821cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e17ba01511704ddcbd0e010a716821cd" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44213 } } }
I20260807 03:10:50.218642 29493 sys_catalog.cc:458] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [sys.catalog]: This master's current role is: LEADER
I20260807 03:10:50.218642 29492 sys_catalog.cc:458] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd [sys.catalog]: This master's current role is: LEADER
I20260807 03:10:50.220528 29500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:10:50.229065 29500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:10:50.230083 29485 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 03:10:50.250197 29500 catalog_manager.cc:1347] Generated new cluster ID: 4e06fd1ba2d24af697ab3db0ee91b23c
I20260807 03:10:50.250583 29500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:10:50.277621 29500 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:10:50.279879 29500 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:10:50.289201 29500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd: Generated new TSK 0
I20260807 03:10:50.290088 29500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:10:51.141753 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.31.20:54919/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44213
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 03:10:51 dist-test-slave-tm1g krb5kdc[28948](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786072251, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 03:10:51.728111 29533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:10:51.728498 29533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:10:51.728801 29533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:10:51.740836 29533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:10:51.741123 29533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:10:51.741500 29533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:10:51.758569 29533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.17.31.20:54919/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:10:51.760068 29533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:10:51.761538 29533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:10:51.779618 29545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:51.779654 29546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:51.780671 29548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:10:51.781224 29533 server_base.cc:1034] running on GCE node
I20260807 03:10:51.781802 29533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:10:51.782953 29533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:10:51.784335 29533 hybrid_clock.cc:648] HybridClock initialized: now 1786072251784205 us; error 79 us; skew 500 ppm
I20260807 03:10:51.784765 29533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:10:51 dist-test-slave-tm1g krb5kdc[28948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072251, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:10:51.792552 29533 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:10:51.794618 29533 webserver.cc:460] Webserver started at http://127.17.31.1:37449/ using document root <none> and password file <none>
I20260807 03:10:51.795430 29533 fs_manager.cc:359] Metadata directory not provided
I20260807 03:10:51.795605 29533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:10:51.795982 29533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 03:10:51 dist-test-slave-tm1g krb5kdc[28948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072251, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
I20260807 03:10:51.832986 29533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b"
format_stamp: "Formatted at 2026-08-07 03:10:51 on dist-test-slave-tm1g"
server_key: "e88c02a7008df20b968058d0d87c3d53"
server_key_iv: "cb9909231538e976a84aadfc9e6d6ca8"
server_key_version: "kuduclusterkey@0"
I20260807 03:10:51.834499 29533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b"
format_stamp: "Formatted at 2026-08-07 03:10:51 on dist-test-slave-tm1g"
server_key: "e88c02a7008df20b968058d0d87c3d53"
server_key_iv: "cb9909231538e976a84aadfc9e6d6ca8"
server_key_version: "kuduclusterkey@0"
I20260807 03:10:51.845552 29533 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.004s
I20260807 03:10:51.883549 29556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:51.885818 29533 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.006s sys 0.010s
I20260807 03:10:51.886125 29533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b"
format_stamp: "Formatted at 2026-08-07 03:10:51 on dist-test-slave-tm1g"
server_key: "e88c02a7008df20b968058d0d87c3d53"
server_key_iv: "cb9909231538e976a84aadfc9e6d6ca8"
server_key_version: "kuduclusterkey@0"
I20260807 03:10:51.886569 29533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:10:51.932066 29533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:10:51.938542 29533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:10:51.939054 29533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:10:51.946777 29533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:10:51.949182 29533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:10:51.949316 29533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:51.949534 29533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:10:51.949638 29533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 03:10:52 dist-test-slave-tm1g krb5kdc[28948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072251, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
I20260807 03:10:52.235616 29533 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:44181
I20260807 03:10:52.235669 29676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:44181 every 8 connection(s)
I20260807 03:10:52.237360 29533 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:10:52.246863 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 29533
I20260807 03:10:52.247126 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:10:52.248394 17532 external_mini_cluster.cc:1428] Setting key c2a6288d2aa7d821bcaa72faf2561779
Aug 07 03:10:52 dist-test-slave-tm1g krb5kdc[28948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072251, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:10:52.304930 29679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:52.242466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44213 (local address 127.17.31.1:52907)
0807 03:10:52.264774 (+ 22308us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:52.264848 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:52.268389 (+ 3541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:52.269555 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:52.269567 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:52.270981 (+ 1414us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:10:52.272276 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:52.272302 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:52.274511 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:52.274524 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:52.281595 (+ 7071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:52.281611 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:52.284172 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:52.284181 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:52.284566 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:52.291653 (+ 7087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:10:52.291688 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:10:52.295647 (+ 3959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:10:52.300025 (+ 4378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:10:52.300035 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:10:52.300052 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:10:52.300490 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:10:52.300941 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:10:52.300946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:10:52.300950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:10:52.301147 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:10:52.301880 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:10:52.301892 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:10:52.302222 (+ 330us) client_negotiation.cc:769] Sending connection context
0807 03:10:52.302593 (+ 371us) client_negotiation.cc:240] Negotiation successful
0807 03:10:52.303258 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21896,"thread_start_us":21431,"threads_started":1}
I20260807 03:10:52.306370 29680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:52.243560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:52907 (local address 127.17.31.62:44213)
0807 03:10:52.263506 (+ 19946us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:52.263510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:52.264997 (+ 1487us) server_negotiation.cc:407] Connection header received
0807 03:10:52.268891 (+ 3894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:52.268897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:52.268979 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:52.269107 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:52.272480 (+ 3373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:52.274169 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:52.282033 (+ 7864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:52.283332 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:52.283564 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:52.295959 (+ 12395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:52.295969 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:52.295973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:10:52.296027 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:52.299884 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:52.300621 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:52.300627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:52.300630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:52.300824 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:52.301270 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:52.301279 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:52.301281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:52.301623 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:52.301800 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:52.305495 (+ 3695us) server_negotiation.cc:299] Negotiation successful
0807 03:10:52.305779 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19798,"thread_start_us":19665,"threads_started":1}
I20260807 03:10:52.308656 29677 heartbeater.cc:347] Connected to a master server at 127.17.31.62:44213
I20260807 03:10:52.308996 29677 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
I20260807 03:10:52.309700 29677 heartbeater.cc:511] Master 127.17.31.62:44213 requested a full tablet report, sending...
I20260807 03:10:52.313777 29446 ts_manager.cc:194] Registered new tserver with Master: 1e06a4cb778a4cf3b8a1b5cee624e72b (127.17.31.1:44181)
I20260807 03:10:52.315845 29446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:52907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:10:52.359769 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.31.20:54919/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44213
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 03:10:52 dist-test-slave-tm1g krb5kdc[28948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072251, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
W20260807 03:10:53.075838 29684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:10:53.076112 29684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:10:53.076210 29684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:10:53.085165 29684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:10:53.085372 29684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:10:53.085551 29684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:10:53.098652 29684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.17.31.20:54919/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:10:53.100081 29684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:10:53.101555 29684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:10:53.120913 29700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:53.120826 29701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:53.121150 29703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:10:53.122906 29684 server_base.cc:1034] running on GCE node
I20260807 03:10:53.123589 29684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:10:53.124752 29684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:10:53.126050 29684 hybrid_clock.cc:648] HybridClock initialized: now 1786072253125931 us; error 45 us; skew 500 ppm
I20260807 03:10:53.126554 29684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:10:53 dist-test-slave-tm1g krb5kdc[28948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072253, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:10:53.131381 29684 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:10:53.133196 29684 webserver.cc:460] Webserver started at http://127.17.31.2:45893/ using document root <none> and password file <none>
I20260807 03:10:53.134081 29684 fs_manager.cc:359] Metadata directory not provided
I20260807 03:10:53.134274 29684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:10:53.134722 29684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 03:10:53 dist-test-slave-tm1g krb5kdc[28948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072253, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
I20260807 03:10:53.154943 29684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "3a323f3a11c64ea592f5949481cc5c37"
format_stamp: "Formatted at 2026-08-07 03:10:53 on dist-test-slave-tm1g"
server_key: "a0650be8d857ec80f2a329e53fe84e00"
server_key_iv: "86da6af9af80400a4a863c36d7a3b102"
server_key_version: "kuduclusterkey@0"
I20260807 03:10:53.156720 29684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "3a323f3a11c64ea592f5949481cc5c37"
format_stamp: "Formatted at 2026-08-07 03:10:53 on dist-test-slave-tm1g"
server_key: "a0650be8d857ec80f2a329e53fe84e00"
server_key_iv: "86da6af9af80400a4a863c36d7a3b102"
server_key_version: "kuduclusterkey@0"
I20260807 03:10:53.165645 29684 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260807 03:10:53.183980 29711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:53.185705 29684 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260807 03:10:53.185959 29684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "3a323f3a11c64ea592f5949481cc5c37"
format_stamp: "Formatted at 2026-08-07 03:10:53 on dist-test-slave-tm1g"
server_key: "a0650be8d857ec80f2a329e53fe84e00"
server_key_iv: "86da6af9af80400a4a863c36d7a3b102"
server_key_version: "kuduclusterkey@0"
I20260807 03:10:53.186391 29684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:10:53.212428 29684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:10:53.217861 29684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:10:53.218446 29684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:10:53.220378 29684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:10:53.222802 29684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:10:53.222973 29684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:53.223203 29684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:10:53.223388 29684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:53.314990 29829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:39593 every 8 connection(s)
I20260807 03:10:53.315001 29684 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:39593
I20260807 03:10:53.317479 29684 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:10:53.319159 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 29684
I20260807 03:10:53.319340 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:10:53.319856 29677 heartbeater.cc:503] Master 127.17.31.62:44213 was elected leader, sending a full tablet report...
I20260807 03:10:53.320025 17532 external_mini_cluster.cc:1428] Setting key 8a4f21c2f27dc6aad88903cf15c2642a
Aug 07 03:10:53 dist-test-slave-tm1g krb5kdc[28948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072253, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:10:53.348079 29832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:53.321829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:38169 (local address 127.17.31.62:44213)
0807 03:10:53.322759 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:53.322764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:53.324266 (+ 1502us) server_negotiation.cc:407] Connection header received
0807 03:10:53.327356 (+ 3090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:53.327361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:53.327472 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:53.327603 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:53.329565 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:53.330957 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:53.333363 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:53.333952 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:53.334115 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:53.339580 (+ 5465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:53.339590 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:53.339596 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:10:53.339677 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:53.343930 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:53.344913 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:53.344917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:53.344919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:53.345120 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:53.345855 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:53.345858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:53.345860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:53.346271 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:53.346501 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:53.347205 (+ 704us) server_negotiation.cc:299] Negotiation successful
0807 03:10:53.347452 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":601,"threads_started":1}
I20260807 03:10:53.349309 29833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:53.322585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44213 (local address 127.17.31.2:38169)
0807 03:10:53.324105 (+ 1520us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:53.324161 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:53.327104 (+ 2943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:53.327930 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:53.327939 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:53.328625 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:10:53.329398 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:53.329417 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:53.331149 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:53.331153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:53.333221 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:53.333233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:53.334387 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:53.334397 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:53.334793 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:53.335498 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:10:53.335520 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:10:53.339190 (+ 3670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:10:53.344073 (+ 4883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:10:53.344084 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:10:53.344103 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:10:53.344775 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:10:53.345243 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:10:53.345251 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:10:53.345258 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:10:53.345739 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:10:53.346473 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:10:53.346481 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:10:53.346834 (+ 353us) client_negotiation.cc:769] Sending connection context
0807 03:10:53.347109 (+ 275us) client_negotiation.cc:240] Negotiation successful
0807 03:10:53.347737 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1210,"thread_start_us":736,"threads_started":1}
I20260807 03:10:53.351467 29830 heartbeater.cc:347] Connected to a master server at 127.17.31.62:44213
I20260807 03:10:53.351815 29830 heartbeater.cc:464] Registering TS with master...
I20260807 03:10:53.352666 29830 heartbeater.cc:511] Master 127.17.31.62:44213 requested a full tablet report, sending...
I20260807 03:10:53.354817 29445 ts_manager.cc:194] Registered new tserver with Master: 3a323f3a11c64ea592f5949481cc5c37 (127.17.31.2:39593)
I20260807 03:10:53.355844 29445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:38169
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:10:53.388135 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.31.20:54919/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44213
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:10:53.971426 29836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:10:53.971729 29836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:10:53.971942 29836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:10:53.980978 29836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:10:53.981195 29836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:10:53.981454 29836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:10:53.994078 29836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.17.31.20:54919/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:10:53.995596 29836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:10:53.997057 29836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:10:54.014370 29841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:54.014271 29844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:10:54.014554 29842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:10:54.015877 29836 server_base.cc:1034] running on GCE node
I20260807 03:10:54.016583 29836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:10:54.017580 29836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:10:54.018858 29836 hybrid_clock.cc:648] HybridClock initialized: now 1786072254018747 us; error 73 us; skew 500 ppm
I20260807 03:10:54.019332 29836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:10:54 dist-test-slave-tm1g krb5kdc[28948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072254, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:10:54.023837 29836 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:10:54.025643 29836 webserver.cc:460] Webserver started at http://127.17.31.3:40995/ using document root <none> and password file <none>
I20260807 03:10:54.026607 29836 fs_manager.cc:359] Metadata directory not provided
I20260807 03:10:54.026859 29836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:10:54.027302 29836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 03:10:54 dist-test-slave-tm1g krb5kdc[28948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072254, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
I20260807 03:10:54.042562 29836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "41c047a4d3154c429b96878a73b03b41"
format_stamp: "Formatted at 2026-08-07 03:10:54 on dist-test-slave-tm1g"
server_key: "def04d2c1de37324793f8d60602333da"
server_key_iv: "9eb43a0a59bb27300f742fcd51016e0d"
server_key_version: "kuduclusterkey@0"
I20260807 03:10:54.043697 29836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "41c047a4d3154c429b96878a73b03b41"
format_stamp: "Formatted at 2026-08-07 03:10:54 on dist-test-slave-tm1g"
server_key: "def04d2c1de37324793f8d60602333da"
server_key_iv: "9eb43a0a59bb27300f742fcd51016e0d"
server_key_version: "kuduclusterkey@0"
I20260807 03:10:54.050768 29836 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260807 03:10:54.067528 29851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:54.069068 29836 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260807 03:10:54.069283 29836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "41c047a4d3154c429b96878a73b03b41"
format_stamp: "Formatted at 2026-08-07 03:10:54 on dist-test-slave-tm1g"
server_key: "def04d2c1de37324793f8d60602333da"
server_key_iv: "9eb43a0a59bb27300f742fcd51016e0d"
server_key_version: "kuduclusterkey@0"
I20260807 03:10:54.069556 29836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:10:54.094187 29836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:10:54.099327 29836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:10:54.099817 29836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:10:54.101075 29836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:10:54.103644 29836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:10:54.103847 29836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:54.104082 29836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:10:54.104262 29836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:10:54.200785 29836 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:44523
I20260807 03:10:54.200814 29977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:44523 every 8 connection(s)
I20260807 03:10:54.202425 29836 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:10:54.205137 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 29836
I20260807 03:10:54.205323 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:10:54.205807 17532 external_mini_cluster.cc:1428] Setting key f4da670637c9590e5315a74a4a0919f0
Aug 07 03:10:54 dist-test-slave-tm1g krb5kdc[28948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072254, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:10:54.230711 29980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.207060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:59741 (local address 127.17.31.62:44213)
0807 03:10:54.207874 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:54.207878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:54.209611 (+ 1733us) server_negotiation.cc:407] Connection header received
0807 03:10:54.212581 (+ 2970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:54.212587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:54.212725 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:54.212844 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:54.215004 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.216085 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.218141 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.218684 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.218816 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.223584 (+ 4768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:54.223597 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:54.223601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:10:54.223643 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:54.226952 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:54.227830 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:54.227841 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:54.227845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:54.228005 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:54.228567 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:54.228574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:54.228580 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:54.228958 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:54.229122 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:54.229864 (+ 742us) server_negotiation.cc:299] Negotiation successful
0807 03:10:54.230108 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":598,"threads_started":1}
I20260807 03:10:54.231808 29983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.207990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44213 (local address 127.17.31.3:59741)
0807 03:10:54.209450 (+ 1460us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:54.209495 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:54.212308 (+ 2813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:54.213161 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:54.213169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:54.213839 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:10:54.214835 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:54.214855 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.216276 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.216280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:54.217969 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:54.217982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.219089 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.219098 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:54.219337 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.219983 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:10:54.220003 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:10:54.223159 (+ 3156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:10:54.227097 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:10:54.227105 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:10:54.227120 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:10:54.227693 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:10:54.228123 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:10:54.228127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:10:54.228131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:10:54.228430 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:10:54.229088 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:10:54.229096 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:10:54.229419 (+ 323us) client_negotiation.cc:769] Sending connection context
0807 03:10:54.229711 (+ 292us) client_negotiation.cc:240] Negotiation successful
0807 03:10:54.230404 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"thread_start_us":823,"threads_started":1}
I20260807 03:10:54.233441 29978 heartbeater.cc:347] Connected to a master server at 127.17.31.62:44213
I20260807 03:10:54.233883 29978 heartbeater.cc:464] Registering TS with master...
I20260807 03:10:54.235136 29978 heartbeater.cc:511] Master 127.17.31.62:44213 requested a full tablet report, sending...
I20260807 03:10:54.236874 29445 ts_manager.cc:194] Registered new tserver with Master: 41c047a4d3154c429b96878a73b03b41 (127.17.31.3:44523)
I20260807 03:10:54.237774 29445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:59741
I20260807 03:10:54.247478 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:10:54.270082 29980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.253899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.17.31.62:44213)
0807 03:10:54.254161 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:54.254187 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:54.255137 (+ 950us) server_negotiation.cc:407] Connection header received
0807 03:10:54.255286 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:54.255289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:54.255390 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:54.255469 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:54.257552 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.258532 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.260156 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.260536 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.260624 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.263658 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:54.263663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:54.263667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:10:54.263695 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:54.267215 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:54.267978 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:54.267982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:54.267984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:54.268164 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:54.268535 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:54.268539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:54.268541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:54.268888 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:54.269140 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:54.269357 (+ 217us) server_negotiation.cc:299] Negotiation successful
0807 03:10:54.269546 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 03:10:54.274838 29445 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:10:54.285497 29445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:10:54.319710 29997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.302786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:44181 (local address 127.0.0.1:56444)
0807 03:10:54.306854 (+ 4068us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:54.306891 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:54.307041 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:54.307923 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:54.307927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:54.307971 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:10:54.309917 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:54.309942 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.312576 (+ 2634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.312581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:54.316498 (+ 3917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:54.316508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.318267 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.318274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:54.318557 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.318611 (+ 54us) client_negotiation.cc:769] Sending connection context
0807 03:10:54.318749 (+ 138us) client_negotiation.cc:240] Negotiation successful
0807 03:10:54.318982 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3848,"thread_start_us":935,"threads_started":1}
I20260807 03:10:54.319868 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.303038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.17.31.1:44181)
0807 03:10:54.304583 (+ 1545us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:54.304591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:54.307171 (+ 2580us) server_negotiation.cc:407] Connection header received
0807 03:10:54.307329 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:54.307336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:54.307660 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:54.307971 (+ 311us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:10:54.310130 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.312386 (+ 2256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.316640 (+ 4254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.318020 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.318163 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.318539 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:54.318853 (+ 314us) server_negotiation.cc:299] Negotiation successful
0807 03:10:54.319059 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1342,"thread_start_us":997,"threads_started":1}
I20260807 03:10:54.320541 29994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.302242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:44523 (local address 127.0.0.1:58012)
0807 03:10:54.304997 (+ 2755us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:54.305077 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:54.305317 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:54.306981 (+ 1664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:54.306988 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:54.307061 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:10:54.308380 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:54.308395 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.311875 (+ 3480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.311882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:54.316039 (+ 4157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:54.316054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.319501 (+ 3447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.319508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:54.319751 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.319784 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 03:10:54.319856 (+ 72us) client_negotiation.cc:240] Negotiation successful
0807 03:10:54.319971 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2412,"mutex_wait_us":25,"spinlock_wait_cycles":44800,"thread_start_us":4090,"threads_started":1}
I20260807 03:10:54.321508 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.302065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58012 (local address 127.17.31.3:44523)
0807 03:10:54.303577 (+ 1512us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:54.303586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:54.305559 (+ 1973us) server_negotiation.cc:407] Connection header received
0807 03:10:54.305665 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:54.305675 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:54.305996 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:54.306243 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:10:54.309157 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.311733 (+ 2576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.316295 (+ 4562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.318662 (+ 2367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.318794 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.319182 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:54.320773 (+ 1591us) server_negotiation.cc:299] Negotiation successful
0807 03:10:54.320974 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1269,"thread_start_us":1353,"threads_started":1}
I20260807 03:10:54.325021 29607 tablet_service.cc:1467] Processing CreateTablet for tablet e2285b5c76a74f639ba88704968b931b (DEFAULT_TABLE table=test-table [id=2f3424402765464fb5f2a07e8510842a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:10:54.327381 29607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2285b5c76a74f639ba88704968b931b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:10:54.326617 29904 tablet_service.cc:1467] Processing CreateTablet for tablet e2285b5c76a74f639ba88704968b931b (DEFAULT_TABLE table=test-table [id=2f3424402765464fb5f2a07e8510842a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:10:54.328581 29904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2285b5c76a74f639ba88704968b931b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:10:54.331044 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.304813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:39593 (local address 127.0.0.1:60366)
0807 03:10:54.309369 (+ 4556us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:54.309386 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:54.309547 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:54.311040 (+ 1493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:54.311045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:54.311070 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:10:54.312226 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:54.312243 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.325352 (+ 13109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.325356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:54.327966 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:54.327983 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.329796 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.329803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:54.330098 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.330121 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 03:10:54.330225 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 03:10:54.330378 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4191,"thread_start_us":3484,"threads_started":1}
I20260807 03:10:54.331228 29999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.305472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.17.31.2:39593)
0807 03:10:54.307278 (+ 1806us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:54.307289 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:54.310412 (+ 3123us) server_negotiation.cc:407] Connection header received
0807 03:10:54.310536 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:54.310543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:54.310796 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:54.311042 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:10:54.312577 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.325114 (+ 12537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.328133 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.329445 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.329598 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.329968 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:54.330266 (+ 298us) server_negotiation.cc:299] Negotiation successful
0807 03:10:54.330540 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1654,"thread_start_us":1524,"threads_started":1}
I20260807 03:10:54.335477 29760 tablet_service.cc:1467] Processing CreateTablet for tablet e2285b5c76a74f639ba88704968b931b (DEFAULT_TABLE table=test-table [id=2f3424402765464fb5f2a07e8510842a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:10:54.337019 29760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2285b5c76a74f639ba88704968b931b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:10:54.350031 30000 tablet_bootstrap.cc:492] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b: Bootstrap starting.
I20260807 03:10:54.354923 30002 tablet_bootstrap.cc:492] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37: Bootstrap starting.
I20260807 03:10:54.357609 30000 tablet_bootstrap.cc:654] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b: Neither blocks nor log segments found. Creating new log.
I20260807 03:10:54.358842 30001 tablet_bootstrap.cc:492] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41: Bootstrap starting.
I20260807 03:10:54.359103 29830 heartbeater.cc:503] Master 127.17.31.62:44213 was elected leader, sending a full tablet report...
I20260807 03:10:54.360860 30000 log.cc:824] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b: Log is configured to *not* fsync() on all Append() calls
I20260807 03:10:54.362356 30002 tablet_bootstrap.cc:654] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37: Neither blocks nor log segments found. Creating new log.
I20260807 03:10:54.365168 30002 log.cc:824] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37: Log is configured to *not* fsync() on all Append() calls
I20260807 03:10:54.365793 30001 tablet_bootstrap.cc:654] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41: Neither blocks nor log segments found. Creating new log.
I20260807 03:10:54.367420 30001 log.cc:824] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41: Log is configured to *not* fsync() on all Append() calls
I20260807 03:10:54.367875 30000 tablet_bootstrap.cc:492] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b: No bootstrap required, opened a new log
I20260807 03:10:54.368193 30000 ts_tablet_manager.cc:1397] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.004s
I20260807 03:10:54.369670 30002 tablet_bootstrap.cc:492] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37: No bootstrap required, opened a new log
I20260807 03:10:54.370141 30002 ts_tablet_manager.cc:1397] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.010s
I20260807 03:10:54.371587 30001 tablet_bootstrap.cc:492] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41: No bootstrap required, opened a new log
I20260807 03:10:54.371970 30001 ts_tablet_manager.cc:1397] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260807 03:10:54.385056 30001 raft_consensus.cc:348] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c047a4d3154c429b96878a73b03b41" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44523 } } peers { permanent_uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44181 } } peers { permanent_uuid: "3a323f3a11c64ea592f5949481cc5c37" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 39593 } }
I20260807 03:10:54.385619 30001 raft_consensus.cc:374] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:10:54.385334 30000 raft_consensus.cc:348] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c047a4d3154c429b96878a73b03b41" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44523 } } peers { permanent_uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44181 } } peers { permanent_uuid: "3a323f3a11c64ea592f5949481cc5c37" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 39593 } }
I20260807 03:10:54.385847 30001 raft_consensus.cc:729] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c047a4d3154c429b96878a73b03b41, State: Initialized, Role: FOLLOWER
I20260807 03:10:54.386009 30000 raft_consensus.cc:374] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:10:54.386157 30000 raft_consensus.cc:729] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e06a4cb778a4cf3b8a1b5cee624e72b, State: Initialized, Role: FOLLOWER
I20260807 03:10:54.386718 30001 consensus_queue.cc:260] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c047a4d3154c429b96878a73b03b41" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44523 } } peers { permanent_uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44181 } } peers { permanent_uuid: "3a323f3a11c64ea592f5949481cc5c37" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 39593 } }
I20260807 03:10:54.386682 30002 raft_consensus.cc:348] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c047a4d3154c429b96878a73b03b41" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44523 } } peers { permanent_uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44181 } } peers { permanent_uuid: "3a323f3a11c64ea592f5949481cc5c37" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 39593 } }
I20260807 03:10:54.387293 30002 raft_consensus.cc:374] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:10:54.387007 30000 consensus_queue.cc:260] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c047a4d3154c429b96878a73b03b41" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44523 } } peers { permanent_uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44181 } } peers { permanent_uuid: "3a323f3a11c64ea592f5949481cc5c37" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 39593 } }
I20260807 03:10:54.387595 30002 raft_consensus.cc:729] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a323f3a11c64ea592f5949481cc5c37, State: Initialized, Role: FOLLOWER
I20260807 03:10:54.388558 29978 heartbeater.cc:503] Master 127.17.31.62:44213 was elected leader, sending a full tablet report...
I20260807 03:10:54.388641 30002 consensus_queue.cc:260] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c047a4d3154c429b96878a73b03b41" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44523 } } peers { permanent_uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44181 } } peers { permanent_uuid: "3a323f3a11c64ea592f5949481cc5c37" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 39593 } }
I20260807 03:10:54.394249 30000 ts_tablet_manager.cc:1428] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b: Time spent starting tablet: real 0.026s user 0.008s sys 0.017s
I20260807 03:10:54.395798 30001 ts_tablet_manager.cc:1428] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260807 03:10:54.397248 30002 ts_tablet_manager.cc:1428] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37: Time spent starting tablet: real 0.027s user 0.016s sys 0.007s
W20260807 03:10:54.455060 29979 tablet.cc:2369] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:10:54.495096 30007 raft_consensus.cc:482] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:10:54.495487 30007 raft_consensus.cc:504] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c047a4d3154c429b96878a73b03b41" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44523 } } peers { permanent_uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44181 } } peers { permanent_uuid: "3a323f3a11c64ea592f5949481cc5c37" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 39593 } }
I20260807 03:10:54.497437 30007 leader_election.cc:288] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c047a4d3154c429b96878a73b03b41 (127.17.31.3:44523), 3a323f3a11c64ea592f5949481cc5c37 (127.17.31.2:39593)
I20260807 03:10:54.507102 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.497896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:43579 (local address 127.17.31.3:44523)
0807 03:10:54.498093 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:54.498098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:54.499331 (+ 1233us) server_negotiation.cc:407] Connection header received
0807 03:10:54.499545 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:54.499552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:54.499640 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:54.499730 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:10:54.501046 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.501901 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.504640 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.505813 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.505965 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.506143 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:54.506603 (+ 460us) server_negotiation.cc:299] Negotiation successful
0807 03:10:54.506732 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 03:10:54.507324 30009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.497834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:44523 (local address 127.17.31.1:43579)
0807 03:10:54.499208 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:54.499225 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:54.499394 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:54.499965 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:54.499968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:54.500040 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:10:54.500871 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:54.500881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.502057 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.502063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:54.504461 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:54.504473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.506194 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.506206 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:54.506435 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.506461 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 03:10:54.506576 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 03:10:54.506703 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1201,"thread_start_us":849,"threads_started":1}
I20260807 03:10:54.508574 29999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.497904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:35989 (local address 127.17.31.2:39593)
0807 03:10:54.498253 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:54.498261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:54.501035 (+ 2774us) server_negotiation.cc:407] Connection header received
0807 03:10:54.501230 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:54.501237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:54.501300 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:54.501393 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:10:54.502375 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.503506 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.505939 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.507373 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.507575 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.507773 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:54.508141 (+ 368us) server_negotiation.cc:299] Negotiation successful
0807 03:10:54.508265 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260807 03:10:54.508682 30011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.499283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:39593 (local address 127.17.31.1:35989)
0807 03:10:54.500914 (+ 1631us) negotiation.cc:107] Waiting for socket to connect
0807 03:10:54.500940 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 03:10:54.501087 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:10:54.501409 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:10:54.501413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:10:54.501437 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:10:54.502137 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:54.502147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.503669 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.503675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:54.505735 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:10:54.505747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.507801 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.507806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:10:54.507965 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.507987 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 03:10:54.508104 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 03:10:54.508227 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1504,"thread_start_us":1197,"threads_started":1}
I20260807 03:10:54.508755 29928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2285b5c76a74f639ba88704968b931b" candidate_uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c047a4d3154c429b96878a73b03b41" is_pre_election: true
I20260807 03:10:54.509429 29928 raft_consensus.cc:2454] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e06a4cb778a4cf3b8a1b5cee624e72b in term 0.
I20260807 03:10:54.510375 29559 leader_election.cc:302] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e06a4cb778a4cf3b8a1b5cee624e72b, 41c047a4d3154c429b96878a73b03b41; no voters:
I20260807 03:10:54.510875 30007 raft_consensus.cc:2781] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:10:54.511089 30007 raft_consensus.cc:482] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:10:54.511212 30007 raft_consensus.cc:3037] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:10:54.511271 29783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2285b5c76a74f639ba88704968b931b" candidate_uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a323f3a11c64ea592f5949481cc5c37" is_pre_election: true
I20260807 03:10:54.511986 29783 raft_consensus.cc:2454] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e06a4cb778a4cf3b8a1b5cee624e72b in term 0.
I20260807 03:10:54.515105 30007 raft_consensus.cc:504] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c047a4d3154c429b96878a73b03b41" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44523 } } peers { permanent_uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44181 } } peers { permanent_uuid: "3a323f3a11c64ea592f5949481cc5c37" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 39593 } }
I20260807 03:10:54.516110 30007 leader_election.cc:288] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [CANDIDATE]: Term 1 election: Requested vote from peers 41c047a4d3154c429b96878a73b03b41 (127.17.31.3:44523), 3a323f3a11c64ea592f5949481cc5c37 (127.17.31.2:39593)
I20260807 03:10:54.516528 29928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2285b5c76a74f639ba88704968b931b" candidate_uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c047a4d3154c429b96878a73b03b41"
I20260807 03:10:54.516717 29783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2285b5c76a74f639ba88704968b931b" candidate_uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a323f3a11c64ea592f5949481cc5c37"
I20260807 03:10:54.516868 29928 raft_consensus.cc:3037] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:10:54.517061 29783 raft_consensus.cc:3037] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:10:54.522220 29783 raft_consensus.cc:2454] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e06a4cb778a4cf3b8a1b5cee624e72b in term 1.
I20260807 03:10:54.522184 29928 raft_consensus.cc:2454] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e06a4cb778a4cf3b8a1b5cee624e72b in term 1.
I20260807 03:10:54.522725 29559 leader_election.cc:302] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e06a4cb778a4cf3b8a1b5cee624e72b, 41c047a4d3154c429b96878a73b03b41; no voters:
I20260807 03:10:54.523115 30007 raft_consensus.cc:2781] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:10:54.524096 30007 raft_consensus.cc:686] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [term 1 LEADER]: Becoming Leader. State: Replica: 1e06a4cb778a4cf3b8a1b5cee624e72b, State: Running, Role: LEADER
I20260807 03:10:54.524784 30007 consensus_queue.cc:237] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c047a4d3154c429b96878a73b03b41" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44523 } } peers { permanent_uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44181 } } peers { permanent_uuid: "3a323f3a11c64ea592f5949481cc5c37" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 39593 } }
I20260807 03:10:54.530783 29445 catalog_manager.cc:5495] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e06a4cb778a4cf3b8a1b5cee624e72b (127.17.31.1). New cstate: current_term: 1 leader_uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c047a4d3154c429b96878a73b03b41" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e06a4cb778a4cf3b8a1b5cee624e72b" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a323f3a11c64ea592f5949481cc5c37" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 39593 } health_report { overall_health: UNKNOWN } } }
W20260807 03:10:54.543294 29678 tablet.cc:2369] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:10:54.555907 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.548745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.17.31.1:44181)
0807 03:10:54.549007 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:54.549012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:54.549029 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:10:54.549099 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:54.549102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:54.549212 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:54.549338 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:10:54.549947 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.551122 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.553365 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.554051 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.554145 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.554878 (+ 733us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:10:54.555385 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:10:54.555495 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:54.555575 (+ 80us) server_negotiation.cc:299] Negotiation successful
0807 03:10:54.555634 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 03:10:54.565941 29783 raft_consensus.cc:1261] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37 [term 1 FOLLOWER]: Refusing update from remote peer 1e06a4cb778a4cf3b8a1b5cee624e72b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:10:54.566104 29928 raft_consensus.cc:1261] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41 [term 1 FOLLOWER]: Refusing update from remote peer 1e06a4cb778a4cf3b8a1b5cee624e72b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:10:54.567286 30012 consensus_queue.cc:1035] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c047a4d3154c429b96878a73b03b41" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 44523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:10:54.567960 30007 consensus_queue.cc:1035] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a323f3a11c64ea592f5949481cc5c37" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 39593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 03:10:54.571686 29831 tablet.cc:2369] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:10:54.584710 30015 mvcc.cc:205] Tried to move back new op lower bound from 7315751954690977792 to 7315751954537570304. Current Snapshot: MvccSnapshot[applied={T|T < 7315751954690977792}]
I20260807 03:10:54.585604 30016 mvcc.cc:205] Tried to move back new op lower bound from 7315751954690977792 to 7315751954537570304. Current Snapshot: MvccSnapshot[applied={T|T < 7315751954690977792}]
I20260807 03:10:54.591450 30017 mvcc.cc:205] Tried to move back new op lower bound from 7315751954690977792 to 7315751954537570304. Current Snapshot: MvccSnapshot[applied={T|T < 7315751954690977792}]
I20260807 03:10:54.605499 29446 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:10:54.605832 29446 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:10:54.610687 29446 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e17ba01511704ddcbd0e010a716821cd: Sending DeleteTablet for 3 replicas of tablet e2285b5c76a74f639ba88704968b931b
I20260807 03:10:54.612078 29904 tablet_service.cc:1514] Processing DeleteTablet for tablet e2285b5c76a74f639ba88704968b931b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:10:54 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:58012
I20260807 03:10:54.612474 29607 tablet_service.cc:1514] Processing DeleteTablet for tablet e2285b5c76a74f639ba88704968b931b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:10:54 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:56444
I20260807 03:10:54.612548 29760 tablet_service.cc:1514] Processing DeleteTablet for tablet e2285b5c76a74f639ba88704968b931b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:10:54 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:60366
I20260807 03:10:54.615119 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 29533
I20260807 03:10:54.628664 30024 tablet_replica.cc:330] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41: stopping tablet replica
I20260807 03:10:54.629329 30024 raft_consensus.cc:2229] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:10:54.630355 30024 raft_consensus.cc:2258] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:10:54.635473 30025 tablet_replica.cc:330] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37: stopping tablet replica
I20260807 03:10:54.636184 30024 ts_tablet_manager.cc:1905] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:10:54.636143 30025 raft_consensus.cc:2229] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:10:54.637313 30025 raft_consensus.cc:2258] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:10:54.639593 30026 tablet_replica.cc:330] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b: stopping tablet replica
I20260807 03:10:54.639632 30025 ts_tablet_manager.cc:1905] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:10:54.640272 30026 raft_consensus.cc:2229] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:10:54.641549 30026 raft_consensus.cc:2258] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:10:54.645397 30024 ts_tablet_manager.cc:1918] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:10:54.645710 30024 log.cc:1196] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/e2285b5c76a74f639ba88704968b931b
I20260807 03:10:54.646389 30026 ts_tablet_manager.cc:1905] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:10:54.646533 30024 ts_tablet_manager.cc:1939] T e2285b5c76a74f639ba88704968b931b P 41c047a4d3154c429b96878a73b03b41: Deleting consensus metadata
I20260807 03:10:54.647284 30025 ts_tablet_manager.cc:1918] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:10:54.647535 30025 log.cc:1196] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/e2285b5c76a74f639ba88704968b931b
I20260807 03:10:54.648183 30025 ts_tablet_manager.cc:1939] T e2285b5c76a74f639ba88704968b931b P 3a323f3a11c64ea592f5949481cc5c37: Deleting consensus metadata
I20260807 03:10:54.648908 29432 catalog_manager.cc:4841] TS 41c047a4d3154c429b96878a73b03b41 (127.17.31.3:44523): tablet e2285b5c76a74f639ba88704968b931b (table test-table [id=2f3424402765464fb5f2a07e8510842a]) successfully deleted
I20260807 03:10:54.650157 29430 catalog_manager.cc:4841] TS 3a323f3a11c64ea592f5949481cc5c37 (127.17.31.2:39593): tablet e2285b5c76a74f639ba88704968b931b (table test-table [id=2f3424402765464fb5f2a07e8510842a]) successfully deleted
Aug 07 03:10:54 dist-test-slave-tm1g krb5kdc[28948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:10:54.659777 30026 ts_tablet_manager.cc:1918] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:10:54.660112 30026 log.cc:1196] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/e2285b5c76a74f639ba88704968b931b
I20260807 03:10:54.661118 30026 ts_tablet_manager.cc:1939] T e2285b5c76a74f639ba88704968b931b P 1e06a4cb778a4cf3b8a1b5cee624e72b: Deleting consensus metadata
I20260807 03:10:54.663700 29430 catalog_manager.cc:4841] TS 1e06a4cb778a4cf3b8a1b5cee624e72b (127.17.31.1:44181): tablet e2285b5c76a74f639ba88704968b931b (table test-table [id=2f3424402765464fb5f2a07e8510842a]) successfully deleted
I20260807 03:10:54.664024 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.617007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.17.31.1:44181)
0807 03:10:54.617218 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:54.617227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:54.631140 (+ 13913us) server_negotiation.cc:407] Connection header received
0807 03:10:54.631445 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:54.631451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:54.631563 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:54.631641 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:54.639300 (+ 7659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.641831 (+ 2531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.646601 (+ 4770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.647278 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.647376 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.654870 (+ 7494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:54.654885 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:54.654898 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:10:54.654966 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:54.660140 (+ 5174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:54.661019 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:54.661034 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:54.661038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:54.661347 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:54.661808 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:54.661813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:54.661816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:54.662292 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:54.662821 (+ 529us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:54.663175 (+ 354us) server_negotiation.cc:299] Negotiation successful
0807 03:10:54.663462 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 03:10:54.665066 29672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:10:54.763655 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 29533
I20260807 03:10:54.782451 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 29684
Aug 07 03:10:54 dist-test-slave-tm1g krb5kdc[28948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:10:54.797151 29999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.783271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60370 (local address 127.17.31.2:39593)
0807 03:10:54.783449 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:54.783454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:54.783472 (+ 18us) server_negotiation.cc:407] Connection header received
0807 03:10:54.783608 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:54.783611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:54.783686 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:54.783770 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:54.785172 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.785937 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.787329 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.787785 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.787877 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.792008 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:54.792019 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:54.792034 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:10:54.792062 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:54.794815 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:54.795475 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:54.795482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:54.795484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:54.795614 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:54.795919 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:54.795922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:54.795924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:54.796254 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:54.796473 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:54.796727 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 03:10:54.796896 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:10:54.797880 29825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:10:54.887733 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 29684
I20260807 03:10:54.906049 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 29836
Aug 07 03:10:54 dist-test-slave-tm1g krb5kdc[28948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:10:54.921510 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:10:54.906913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.17.31.3:44523)
0807 03:10:54.907089 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 03:10:54.907094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:10:54.907113 (+ 19us) server_negotiation.cc:407] Connection header received
0807 03:10:54.907262 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:10:54.907266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:10:54.907332 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:10:54.907441 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:10:54.908681 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.909377 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.910817 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:10:54.911263 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:10:54.911351 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:10:54.915777 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:10:54.915791 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:10:54.915808 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:10:54.915838 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:10:54.918498 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:54.919386 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:54.919393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:54.919395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:54.919534 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:10:54.919901 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:10:54.919913 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:10:54.919917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:10:54.920297 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:10:54.920646 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0807 03:10:54.920952 (+ 306us) server_negotiation.cc:299] Negotiation successful
0807 03:10:54.921179 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 03:10:54.922151 29973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:10:55.010011 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 29836
I20260807 03:10:55.028254 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 29414
I20260807 03:10:55.029433 29480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:10:55.139635 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 29414
I20260807 03:10:55.163221 17532 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 03:10:55.497718 17532 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 03:10:55.498013 17532 mini_ranger.cc:66] Stopping Ranger...
I20260807 03:10:55.843547 17532 mini_ranger.cc:68] Stopped Ranger
2026-08-07 03:10:55.843 UTC [28978] LOG: received smart shutdown request
2026-08-07 03:10:55.847 UTC [28978] LOG: background worker "logical replication launcher" (PID 28988) exited with exit code 1
2026-08-07 03:10:55.847 UTC [28983] LOG: shutting down
2026-08-07 03:10:56.000 UTC [28978] LOG: database system is shut down
2026-08-07T03:10:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49639 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:10:56 dist-test-slave-tm1g krb5kdc[30040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:10:56 dist-test-slave-tm1g krb5kdc[30040](info): set up 2 sockets
Aug 07 03:10:56 dist-test-slave-tm1g krb5kdc[30040](info): commencing operation
krb5kdc: starting...
W20260807 03:10:57.250571 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:10:57 dist-test-slave-tm1g krb5kdc[30040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:10:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:10:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.17.31.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.17.31.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.17.31.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/rangeradmin_127.17.31.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.17.31.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/rangeradmin_127.17.31.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.17.31.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.17.31.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.17.31.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/rangerlookup_127.17.31.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.17.31.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/rangerlookup_127.17.31.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.31.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.17.31.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.31.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/HTTP_127.17.31.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.31.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/HTTP_127.17.31.20@KRBTEST.COM.keytab.
I20260807 03:10:57.560555 17532 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasko9xIn4/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/postgres -l logfile start
2026-08-07 03:10:59.867 UTC [30073] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 03:10:59.867 UTC [30073] LOG: listening on IPv4 address "127.17.31.20", port 55103
2026-08-07 03:10:59.870 UTC [30073] LOG: listening on Unix socket "/tmp/.s.PGSQL.55103"
2026-08-07 03:10:59.892 UTC [30077] LOG: database system was shut down at 2026-08-07 03:10:58 UTC
2026-08-07 03:10:59.895 UTC [30073] LOG: database system is ready to accept connections
I20260807 03:11:00.879912 17532 mini_postgres.cc:96] Postgres bound to 55103
2026-08-07 03:11:00.899 UTC [30085] FATAL: database "slave" does not exist
127.17.31.20:55103 - accepting connections
I20260807 03:11:00.900406 17532 mini_ranger.cc:161] Starting Ranger...
I20260807 03:11:00.927018 17532 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 03:11:01.376838 17532 mini_ranger.cc:87] Created ranger Postgres database
I20260807 03:11:01.376933 17532 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-admin
2026-08-07 03:11:01,665 [I] DB FLAVOR :POSTGRES
2026-08-07 03:11:01,666 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 03:11:01,666 [I] Checking connection..
2026-08-07 03:11:01,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 03:11:01,895 [I] Checking connection passed.
2026-08-07 03:11:01,895 [I] --------- Verifying version history table ---------
2026-08-07 03:11:01,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 03:11:02,150 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 03:11:02,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 03:11:02,417 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 03:11:02,418 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 03:11:02,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 03:11:02.677 UTC [30149] WARNING: there is no transaction in progress
2026-08-07 03:11:02,683 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 03:11:02,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 03:11:02,910 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 03:11:02,910 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 03:11:02,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 03:11:03,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 03:11:03,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tm1g.c.gcp-upstream.internal','N') ;"
2026-08-07 03:11:03,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 03:11:03,906 [I] Table x_portal_user does not exist in database ranger
2026-08-07 03:11:03,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 03:11:04,161 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 03:11:04,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 03:11:04,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 03:11:04,646 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 03:11:04,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 03:11:05.566 UTC [30261] WARNING: there is no transaction in progress
2026-08-07 03:11:05.581 UTC [30261] WARNING: there is no transaction in progress
2026-08-07 03:11:05.596 UTC [30261] WARNING: there is no transaction in progress
2026-08-07 03:11:05.609 UTC [30261] WARNING: there is no transaction in progress
2026-08-07 03:11:05.627 UTC [30261] WARNING: there is no transaction in progress
2026-08-07 03:11:05.645 UTC [30261] WARNING: there is no transaction in progress
2026-08-07 03:11:05.698 UTC [30261] WARNING: there is no transaction in progress
2026-08-07 03:11:05.704 UTC [30261] WARNING: there is no transaction in progress
2026-08-07 03:11:05.717 UTC [30261] WARNING: there is no transaction in progress
2026-08-07 03:11:05.726 UTC [30261] WARNING: there is no transaction in progress
2026-08-07 03:11:05.736 UTC [30261] WARNING: there is no transaction in progress
2026-08-07 03:11:05.749 UTC [30261] WARNING: there is no transaction in progress
2026-08-07 03:11:05.758 UTC [30261] WARNING: there is no transaction in progress
2026-08-07 03:11:05.764 UTC [30261] WARNING: there is no transaction in progress
2026-08-07 03:11:05.770 UTC [30261] WARNING: there is no transaction in progress
2026-08-07 03:11:05.776 UTC [30261] WARNING: there is no transaction in progress
2026-08-07 03:11:06,330 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 03:11:06,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 03:11:06,567 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 03:11:06,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 03:11:06,823 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 03:11:06,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 03:11:07,077 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 03:11:07,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 03:11:07,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 03:11:07,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tm1g.c.gcp-upstream.internal';"
2026-08-07 03:11:07,817 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 03:11:07,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 03:11:08,061 [I] DB_PATCHES have already been applied
I20260807 03:11:08.067188 17532 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-admin:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasko9xIn4/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 03:11:08.067267 17532 mini_ranger.cc:193] Using host: 127.17.31.20
I20260807 03:11:08.084498 17532 mini_ranger.cc:227] Ranger admin URL: http://127.17.31.20:40095
Aug 07, 2026 3:11:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 3:11:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 3:11:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 3:11:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 3:11:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 3:11:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 3:11:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 03:11:09.227878 17532 mini_ranger.cc:160] Time spent starting Ranger: real 8.327s user 0.004s sys 0.040s
Aug 07, 2026 3:11:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 3:11:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 07, 2026 3:11:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 3:11:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 3:11:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 3:11:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 3:11:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 3:11:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 3:11:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 3:11:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 3:11:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 3:11:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 3:11:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 3:11:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 3:11:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 3:11:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 3:11:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 3:11:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 03:11:30.952421 17532 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.17.31.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.17.31.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.17.31.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/rangerkms_127.17.31.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.17.31.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/rangerkms_127.17.31.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.31.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.17.31.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.31.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/HTTP_127.17.31.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.31.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/HTTP_127.17.31.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 03:11:31 dist-test-slave-tm1g krb5kdc[30040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072291, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 03:11:31.134268 17532 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 03:11:31.154552 17532 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 03:11:31.937777 17532 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 03:11:31.937912 17532 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-kms
I20260807 03:11:31.937953 17532 mini_ranger_kms.cc:227] Using postgres at 127.17.31.20:55103
2026-08-07 03:11:32,344 [I] DB FLAVOR :POSTGRES
2026-08-07 03:11:32,345 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 03:11:32,345 [I] Checking connection
2026-08-07 03:11:32,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 03:11:32,601 [I] connection success
2026-08-07 03:11:32,602 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 03:11:32,602 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 03:11:32,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 03:11:32,901 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 03:11:32,901 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 03:11:32,902 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 03:11:32,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.31.20:55103/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 03:11:33,176 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 03:11:33.652379 17532 mini_ranger_kms.cc:310] Created kms service
I20260807 03:11:33.856248 17532 mini_ranger_kms.cc:326] Created kudu user
I20260807 03:11:33.901875 17532 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 03:11:33 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072291, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
I20260807 03:11:34.293255 17532 mini_ranger_kms.cc:384] Added ranger policy
I20260807 03:11:34.293644 17532 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-kms:/tmp/dist-test-tasko9xIn4/build/asan/bin/postgresql.jar:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasko9xIn4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 03:11:34.293732 17532 mini_ranger_kms.cc:240] Using host: 127.17.31.20
I20260807 03:11:34.311547 17532 mini_ranger_kms.cc:276] Ranger KMS PID: 30453
I20260807 03:11:34.311681 17532 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.17.31.20:56181
Aug 07, 2026 3:11:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 3:11:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 3:11:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 3:11:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 3:11:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 3:11:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260807 03:11:36.701246 17532 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.567s user 0.086s sys 0.342s
I20260807 03:11:36.701507 17532 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 03:11:36.701694 17532 mini_ranger_kms.cc:401] 127.17.31.20:56181/kms/v1/keys
Aug 07 03:11:36 dist-test-slave-tm1g krb5kdc[30040](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786072296, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.31.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 07 03:11:37 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072296, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.31.20@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
Aug 07 03:11:37 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072296, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.31.20@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
Aug 07 03:11:39 dist-test-slave-tm1g krb5kdc[30040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:11:40.006361 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:36437
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:36437
--ranger_config_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.31.20:56181/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:11:40.333259 30506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:11:40.333556 30506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:11:40.333712 30506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:11:40.340565 30506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:11:40.340698 30506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:11:40.340802 30506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:11:40.340898 30506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:11:40.340986 30506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:11:40.350631 30506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/wal
--ranger_kms_url=127.17.31.20:56181/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:36437
--ranger_config_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:36437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:11:40.351774 30506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:11:40.353103 30506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:11:40.364910 30512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:40.365011 30511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:40.364866 30514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:40.365935 30506 server_base.cc:1034] running on GCE node
I20260807 03:11:40.366775 30506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:11:40.368062 30506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:11:40.369287 30506 hybrid_clock.cc:648] HybridClock initialized: now 1786072300369256 us; error 53 us; skew 500 ppm
I20260807 03:11:40.369622 30506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:11:40 dist-test-slave-tm1g krb5kdc[30040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072300, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:40.375835 30506 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:11:40.377206 30506 webserver.cc:460] Webserver started at http://127.17.31.62:41825/ using document root <none> and password file <none>
I20260807 03:11:40.377815 30506 fs_manager.cc:359] Metadata directory not provided
I20260807 03:11:40.377911 30506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:11:40.378159 30506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 03:11:40 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072300, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
I20260807 03:11:40.435491 30506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "7024138b70bf4671a96b0b8ac8c99668"
format_stamp: "Formatted at 2026-08-07 03:11:40 on dist-test-slave-tm1g"
server_key: "96a159bafd0922c52d478fd8d451019d"
server_key_iv: "ebdeaf100beb31482e8f3f547801e394"
server_key_version: "kuduclusterkey@0"
I20260807 03:11:40.436290 30506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "7024138b70bf4671a96b0b8ac8c99668"
format_stamp: "Formatted at 2026-08-07 03:11:40 on dist-test-slave-tm1g"
server_key: "96a159bafd0922c52d478fd8d451019d"
server_key_iv: "ebdeaf100beb31482e8f3f547801e394"
server_key_version: "kuduclusterkey@0"
I20260807 03:11:40.441670 30506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260807 03:11:40.460711 30521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:40.462627 30506 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 03:11:40.462811 30506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "7024138b70bf4671a96b0b8ac8c99668"
format_stamp: "Formatted at 2026-08-07 03:11:40 on dist-test-slave-tm1g"
server_key: "96a159bafd0922c52d478fd8d451019d"
server_key_iv: "ebdeaf100beb31482e8f3f547801e394"
server_key_version: "kuduclusterkey@0"
I20260807 03:11:40.463044 30506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:11:40.480309 30506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:11:40.483733 30506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:11:40.484043 30506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:11:40.501938 30506 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:36437
I20260807 03:11:40.501937 30577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:36437 every 8 connection(s)
I20260807 03:11:40.503280 30506 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:11:40.506727 30578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:11:40.512383 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 30506
I20260807 03:11:40.512603 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:11:40.513262 17532 external_mini_cluster.cc:1428] Setting key bc8b7390d72308ef076da5f2fe7b2bb7
I20260807 03:11:40.517490 30578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668: Bootstrap starting.
I20260807 03:11:40.521229 30578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668: Neither blocks nor log segments found. Creating new log.
I20260807 03:11:40.522493 30578 log.cc:824] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668: Log is configured to *not* fsync() on all Append() calls
I20260807 03:11:40.525519 30578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668: No bootstrap required, opened a new log
Aug 07 03:11:40 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:11:40.534768 30578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7024138b70bf4671a96b0b8ac8c99668" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36437 } }
I20260807 03:11:40.535172 30578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:11:40.535285 30578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7024138b70bf4671a96b0b8ac8c99668, State: Initialized, Role: FOLLOWER
I20260807 03:11:40.535961 30578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7024138b70bf4671a96b0b8ac8c99668" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36437 } }
I20260807 03:11:40.536244 30578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:11:40.536402 30578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:11:40.536597 30578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:11:40.537160 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:40.515292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60434 (local address 127.17.31.62:36437)
0807 03:11:40.516103 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:40.516117 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:40.516180 (+ 63us) server_negotiation.cc:407] Connection header received
0807 03:11:40.518585 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:40.518608 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:40.519121 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:40.519542 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:40.520686 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:40.522745 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:40.524329 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:40.525316 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:40.525518 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:40.529223 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:40.529233 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:40.529255 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:40.529293 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:40.533052 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:40.533735 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:40.533744 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:40.533752 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:40.533933 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:40.534266 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:40.534271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:40.534274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:40.534939 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:40.535247 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:40.535731 (+ 484us) server_negotiation.cc:299] Negotiation successful
0807 03:11:40.536091 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":327,"threads_started":1}
I20260807 03:11:40.538985 30578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7024138b70bf4671a96b0b8ac8c99668" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36437 } }
I20260807 03:11:40.539459 30578 leader_election.cc:302] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7024138b70bf4671a96b0b8ac8c99668; no voters:
I20260807 03:11:40.540041 30578 leader_election.cc:288] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:11:40.540138 30583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:11:40.540841 30583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 1 LEADER]: Becoming Leader. State: Replica: 7024138b70bf4671a96b0b8ac8c99668, State: Running, Role: LEADER
I20260807 03:11:40.541407 30583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7024138b70bf4671a96b0b8ac8c99668" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36437 } }
I20260807 03:11:40.542034 30578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:11:40.544306 30585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7024138b70bf4671a96b0b8ac8c99668. Latest consensus state: current_term: 1 leader_uuid: "7024138b70bf4671a96b0b8ac8c99668" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7024138b70bf4671a96b0b8ac8c99668" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36437 } } }
I20260807 03:11:40.544275 30584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7024138b70bf4671a96b0b8ac8c99668" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7024138b70bf4671a96b0b8ac8c99668" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36437 } } }
I20260807 03:11:40.544888 30584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [sys.catalog]: This master's current role is: LEADER
I20260807 03:11:40.546396 30585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [sys.catalog]: This master's current role is: LEADER
I20260807 03:11:40.546500 30588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:11:40.552008 30588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:11:40.552578 30578 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 03:11:40.583666 30588 catalog_manager.cc:1347] Generated new cluster ID: 8678b75e63aa4b0d9a0d716fbbb6ca81
I20260807 03:11:40.583865 30588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:11:40.595803 30588 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:11:40.597718 30588 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:11:40.625567 30588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668: Generated new TSK 0
I20260807 03:11:40.626638 30588 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:11:41.320025 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.31.20:56181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:36437
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 03:11:41 dist-test-slave-tm1g krb5kdc[30040](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786072301, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 03:11:41.723663 30625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:11:41.724020 30625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:11:41.724226 30625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:11:41.735523 30625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:11:41.735713 30625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:11:41.735918 30625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:11:41.750849 30625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.17.31.20:56181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:36437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:11:41.752532 30625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:11:41.754395 30625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:11:41.776153 30635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:41.776319 30636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:41.780012 30638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:41.781262 30625 server_base.cc:1034] running on GCE node
I20260807 03:11:41.781982 30625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:11:41.783254 30625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:11:41.784511 30625 hybrid_clock.cc:648] HybridClock initialized: now 1786072301784456 us; error 40 us; skew 500 ppm
I20260807 03:11:41.784904 30625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:11:41 dist-test-slave-tm1g krb5kdc[30040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072301, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:41.789134 30625 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:11:41.795743 30625 webserver.cc:460] Webserver started at http://127.17.31.1:45023/ using document root <none> and password file <none>
I20260807 03:11:41.796655 30625 fs_manager.cc:359] Metadata directory not provided
I20260807 03:11:41.796869 30625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:11:41.797240 30625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 03:11:41 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072301, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
I20260807 03:11:41.819738 30625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "a65cabb99b5a463c8c0c56692dc3c9d3"
format_stamp: "Formatted at 2026-08-07 03:11:41 on dist-test-slave-tm1g"
server_key: "dd9282e44f808d7fc6927a79636bc483"
server_key_iv: "f21184792277f1eb767498ad7d635cf3"
server_key_version: "kuduclusterkey@0"
I20260807 03:11:41.820974 30625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "a65cabb99b5a463c8c0c56692dc3c9d3"
format_stamp: "Formatted at 2026-08-07 03:11:41 on dist-test-slave-tm1g"
server_key: "dd9282e44f808d7fc6927a79636bc483"
server_key_iv: "f21184792277f1eb767498ad7d635cf3"
server_key_version: "kuduclusterkey@0"
I20260807 03:11:41.843786 30625 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.011s sys 0.007s
I20260807 03:11:41.870721 30645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:41.872650 30625 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.008s sys 0.002s
I20260807 03:11:41.872833 30625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "a65cabb99b5a463c8c0c56692dc3c9d3"
format_stamp: "Formatted at 2026-08-07 03:11:41 on dist-test-slave-tm1g"
server_key: "dd9282e44f808d7fc6927a79636bc483"
server_key_iv: "f21184792277f1eb767498ad7d635cf3"
server_key_version: "kuduclusterkey@0"
I20260807 03:11:41.873080 30625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:11:41.897347 30625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:11:41.912158 30625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:11:41.912523 30625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:11:41.926738 30625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:11:41.939308 30625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:11:41.939424 30625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:41.939563 30625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:11:41.939631 30625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:41.995674 30625 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:34729
I20260807 03:11:41.995714 30764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:34729 every 8 connection(s)
I20260807 03:11:41.997423 30625 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:11:41.999944 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 30625
I20260807 03:11:42.000173 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:11:42.001049 17532 external_mini_cluster.cc:1428] Setting key f7b8a8ce65aaa755ecb850534941eea9
Aug 07 03:11:42 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072301, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:11:42.030265 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:42.001641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:59039 (local address 127.17.31.62:36437)
0807 03:11:42.006614 (+ 4973us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:42.006620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:42.006642 (+ 22us) server_negotiation.cc:407] Connection header received
0807 03:11:42.010147 (+ 3505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:42.010153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:42.010254 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:42.010404 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:42.013762 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:42.015293 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:42.017704 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:42.018370 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:42.018468 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:42.023009 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:42.023015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:42.023019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:42.023066 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:42.026620 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:42.027525 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:42.027530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:42.027534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:42.027682 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:42.028196 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:42.028201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:42.028205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:42.028560 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:42.028681 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:42.029648 (+ 967us) server_negotiation.cc:299] Negotiation successful
0807 03:11:42.029886 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4856,"thread_start_us":9474,"threads_started":1}
I20260807 03:11:42.032744 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:42.001004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:36437 (local address 127.17.31.1:59039)
0807 03:11:42.004022 (+ 3018us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:42.004071 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:42.009645 (+ 5574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:42.012062 (+ 2417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:42.012076 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:42.012711 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:42.013552 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:42.013579 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:42.015538 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:42.015545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:42.017528 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:42.017553 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:42.018806 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:42.018813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:42.019110 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:42.019930 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:42.019963 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:42.022750 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:42.026793 (+ 4043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:42.026802 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:42.026819 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:42.027391 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:42.027824 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:42.027829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:42.027833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:42.028069 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:42.028799 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:42.028807 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:42.029211 (+ 404us) client_negotiation.cc:769] Sending connection context
0807 03:11:42.030823 (+ 1612us) client_negotiation.cc:240] Negotiation successful
0807 03:11:42.031369 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2683,"thread_start_us":9212,"threads_started":1}
I20260807 03:11:42.035502 30765 heartbeater.cc:347] Connected to a master server at 127.17.31.62:36437
I20260807 03:11:42.035868 30765 heartbeater.cc:464] Registering TS with master...
I20260807 03:11:42.036798 30765 heartbeater.cc:511] Master 127.17.31.62:36437 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
I20260807 03:11:42.039499 30538 ts_manager.cc:194] Registered new tserver with Master: a65cabb99b5a463c8c0c56692dc3c9d3 (127.17.31.1:34729)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
I20260807 03:11:42.042932 30538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:59039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:11:42.090124 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.31.20:56181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:36437
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 03:11:42 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072301, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
Aug 07 03:11:42 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072301, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
W20260807 03:11:42.608148 30771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:11:42.608628 30771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:11:42.608814 30771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:11:42.620090 30771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:11:42.620288 30771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:11:42.620517 30771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:11:42.635560 30771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.17.31.20:56181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:36437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:11:42.637428 30771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:11:42.639161 30771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:11:42.677996 30779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:42.679822 30782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:42.679960 30778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:42.680361 30771 server_base.cc:1034] running on GCE node
I20260807 03:11:42.681321 30771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:11:42.687914 30771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:11:42.689193 30771 hybrid_clock.cc:648] HybridClock initialized: now 1786072302689088 us; error 97 us; skew 500 ppm
I20260807 03:11:42.689612 30771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:11:42 dist-test-slave-tm1g krb5kdc[30040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072302, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:42.696663 30771 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:11:42.710816 30771 webserver.cc:460] Webserver started at http://127.17.31.2:36269/ using document root <none> and password file <none>
I20260807 03:11:42.711683 30771 fs_manager.cc:359] Metadata directory not provided
I20260807 03:11:42.711792 30771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:11:42.712155 30771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 03:11:42 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072302, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
I20260807 03:11:42.730160 30771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "02f6dcef6a064f56b054922e20a7de72"
format_stamp: "Formatted at 2026-08-07 03:11:42 on dist-test-slave-tm1g"
server_key: "590cea6c5c34726d7247da3c31fb8dd2"
server_key_iv: "98d961126ad07267ceeb9a508312a010"
server_key_version: "kuduclusterkey@0"
I20260807 03:11:42.731251 30771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "02f6dcef6a064f56b054922e20a7de72"
format_stamp: "Formatted at 2026-08-07 03:11:42 on dist-test-slave-tm1g"
server_key: "590cea6c5c34726d7247da3c31fb8dd2"
server_key_iv: "98d961126ad07267ceeb9a508312a010"
server_key_version: "kuduclusterkey@0"
I20260807 03:11:42.738198 30771 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260807 03:11:42.757406 30796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:42.759266 30771 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260807 03:11:42.759434 30771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "02f6dcef6a064f56b054922e20a7de72"
format_stamp: "Formatted at 2026-08-07 03:11:42 on dist-test-slave-tm1g"
server_key: "590cea6c5c34726d7247da3c31fb8dd2"
server_key_iv: "98d961126ad07267ceeb9a508312a010"
server_key_version: "kuduclusterkey@0"
I20260807 03:11:42.759690 30771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:11:42.782982 30771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:11:42.787027 30771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:11:42.787333 30771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:11:42.788364 30771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:11:42.789772 30771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:11:42.789844 30771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:42.789966 30771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:11:42.790032 30771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:42.825392 30771 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:33999
I20260807 03:11:42.825441 30913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:33999 every 8 connection(s)
I20260807 03:11:42.826841 30771 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:11:42.836054 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 30771
I20260807 03:11:42.836243 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:11:42.836715 17532 external_mini_cluster.cc:1428] Setting key 7326c046761e5847586df0161bd1a7f8
Aug 07 03:11:42 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072302, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:11:42.846378 30916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:42.829296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:49323 (local address 127.17.31.62:36437)
0807 03:11:42.829757 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:42.829761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:42.830567 (+ 806us) server_negotiation.cc:407] Connection header received
0807 03:11:42.832437 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:42.832442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:42.832543 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:42.832663 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:42.834021 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:42.834852 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:42.836491 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:42.836921 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:42.837051 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:42.840731 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:42.840735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:42.840739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:42.840768 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:42.843334 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:42.844088 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:42.844095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:42.844099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:42.844225 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:42.844665 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:42.844670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:42.844673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:42.844948 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:42.845100 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:42.845752 (+ 652us) server_negotiation.cc:299] Negotiation successful
0807 03:11:42.845981 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":248,"threads_started":1}
I20260807 03:11:42.847368 30917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:42.829672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:36437 (local address 127.17.31.2:49323)
0807 03:11:42.830419 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:42.830457 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:42.832170 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:42.832896 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:42.832904 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:42.833304 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:42.833843 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:42.833859 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:42.835014 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:42.835024 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:42.836330 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:42.836361 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:42.837143 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:42.837151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:42.837429 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:42.838135 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:42.838163 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:42.840516 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:42.843532 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:42.843540 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:42.843562 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:42.843961 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:42.844333 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:42.844338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:42.844342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:42.844543 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:42.845070 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:42.845077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:42.845401 (+ 324us) client_negotiation.cc:769] Sending connection context
0807 03:11:42.845668 (+ 267us) client_negotiation.cc:240] Negotiation successful
0807 03:11:42.846161 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":280,"threads_started":1}
I20260807 03:11:42.848873 30914 heartbeater.cc:347] Connected to a master server at 127.17.31.62:36437
I20260807 03:11:42.849160 30914 heartbeater.cc:464] Registering TS with master...
I20260807 03:11:42.849808 30914 heartbeater.cc:511] Master 127.17.31.62:36437 requested a full tablet report, sending...
I20260807 03:11:42.851303 30538 ts_manager.cc:194] Registered new tserver with Master: 02f6dcef6a064f56b054922e20a7de72 (127.17.31.2:33999)
I20260807 03:11:42.852133 30538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:49323
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:11:42.890395 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.31.20:56181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:36437
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:11:43.046641 30765 heartbeater.cc:503] Master 127.17.31.62:36437 was elected leader, sending a full tablet report...
W20260807 03:11:43.214637 30920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:11:43.214874 30920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:11:43.214959 30920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:11:43.221563 30920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:11:43.221648 30920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:11:43.221746 30920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:11:43.230754 30920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.17.31.20:56181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:36437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:11:43.231859 30920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:11:43.233029 30920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:11:43.245368 30925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:43.245355 30928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:43.245391 30926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:43.246213 30920 server_base.cc:1034] running on GCE node
I20260807 03:11:43.246852 30920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:11:43.247817 30920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:11:43.249022 30920 hybrid_clock.cc:648] HybridClock initialized: now 1786072303248974 us; error 47 us; skew 500 ppm
I20260807 03:11:43.249361 30920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:11:43 dist-test-slave-tm1g krb5kdc[30040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072303, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:43.252357 30920 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:11:43.253589 30920 webserver.cc:460] Webserver started at http://127.17.31.3:33235/ using document root <none> and password file <none>
I20260807 03:11:43.254163 30920 fs_manager.cc:359] Metadata directory not provided
I20260807 03:11:43.254259 30920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:11:43.254570 30920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 03:11:43 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072303, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
I20260807 03:11:43.269327 30920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "7f05876e7f6b4af98735aaa04d253392"
format_stamp: "Formatted at 2026-08-07 03:11:43 on dist-test-slave-tm1g"
server_key: "f24186349986976c38cbb297fa883870"
server_key_iv: "eaaafc26ca83ee03db6beb3d2cf46778"
server_key_version: "kuduclusterkey@0"
I20260807 03:11:43.270293 30920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "7f05876e7f6b4af98735aaa04d253392"
format_stamp: "Formatted at 2026-08-07 03:11:43 on dist-test-slave-tm1g"
server_key: "f24186349986976c38cbb297fa883870"
server_key_iv: "eaaafc26ca83ee03db6beb3d2cf46778"
server_key_version: "kuduclusterkey@0"
I20260807 03:11:43.276574 30920 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260807 03:11:43.291810 30935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:43.293166 30920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 03:11:43.293347 30920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "7f05876e7f6b4af98735aaa04d253392"
format_stamp: "Formatted at 2026-08-07 03:11:43 on dist-test-slave-tm1g"
server_key: "f24186349986976c38cbb297fa883870"
server_key_iv: "eaaafc26ca83ee03db6beb3d2cf46778"
server_key_version: "kuduclusterkey@0"
I20260807 03:11:43.293581 30920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:11:43.308197 30920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:11:43.311950 30920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:11:43.312250 30920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:11:43.313141 30920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:11:43.314546 30920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:11:43.314618 30920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:43.314742 30920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:11:43.314795 30920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:43.365037 30920 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:41715
I20260807 03:11:43.365041 31055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:41715 every 8 connection(s)
I20260807 03:11:43.366447 30920 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:11:43.373128 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 30920
I20260807 03:11:43.373335 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:11:43.374001 17532 external_mini_cluster.cc:1428] Setting key d86bac1eb3acbd4612e198bdd0a2125a
Aug 07 03:11:43 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072303, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:11:43.391031 31058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:43.369159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:57951 (local address 127.17.31.62:36437)
0807 03:11:43.369708 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:43.369716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:43.370824 (+ 1108us) server_negotiation.cc:407] Connection header received
0807 03:11:43.372808 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:43.372817 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:43.372950 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:43.373088 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:43.375012 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:43.376461 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:43.378323 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:43.379187 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:43.379357 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:43.382953 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:43.382962 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:43.382970 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:43.383016 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:43.386980 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:43.387851 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:43.387860 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:43.387870 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:43.388068 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:43.388580 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:43.388585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:43.388589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:43.388962 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:43.389132 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:43.390070 (+ 938us) server_negotiation.cc:299] Negotiation successful
0807 03:11:43.390443 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":232,"threads_started":1}
I20260807 03:11:43.391391 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:43.369725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:36437 (local address 127.17.31.3:57951)
0807 03:11:43.370651 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:43.370694 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:43.372527 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:43.373351 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:43.373363 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:43.373969 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:43.374785 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:43.374828 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:43.376647 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:43.376652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:43.378115 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:43.378133 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:43.379420 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:43.379425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:43.379609 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:43.380295 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:43.380314 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:43.382725 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:43.387112 (+ 4387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:43.387118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:43.387129 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:43.387690 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:43.388194 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:43.388199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:43.388202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:43.388457 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:43.389205 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:43.389212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:43.389628 (+ 416us) client_negotiation.cc:769] Sending connection context
0807 03:11:43.389863 (+ 235us) client_negotiation.cc:240] Negotiation successful
0807 03:11:43.390223 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":353,"threads_started":1}
I20260807 03:11:43.392786 31056 heartbeater.cc:347] Connected to a master server at 127.17.31.62:36437
I20260807 03:11:43.393035 31056 heartbeater.cc:464] Registering TS with master...
I20260807 03:11:43.393648 31056 heartbeater.cc:511] Master 127.17.31.62:36437 requested a full tablet report, sending...
I20260807 03:11:43.395082 30538 ts_manager.cc:194] Registered new tserver with Master: 7f05876e7f6b4af98735aaa04d253392 (127.17.31.3:41715)
I20260807 03:11:43.396482 30538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:57951
I20260807 03:11:43.401978 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:11:43.404999 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 30625
Aug 07 03:11:43 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:11:43.426055 31061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:43.406063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.17.31.1:34729)
0807 03:11:43.406793 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:43.406804 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:43.407168 (+ 364us) server_negotiation.cc:407] Connection header received
0807 03:11:43.407290 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:43.407301 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:43.407583 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:43.407758 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:43.409272 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:43.411251 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:43.412616 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:43.413653 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:43.413797 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:43.418334 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:43.418352 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:43.418373 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:43.418425 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:43.422806 (+ 4381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:43.423532 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:43.423543 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:43.423549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:43.423765 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:43.424146 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:43.424153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:43.424158 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:43.424677 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:43.425013 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:43.425179 (+ 166us) server_negotiation.cc:299] Negotiation successful
0807 03:11:43.425511 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":338,"threads_started":1}
I20260807 03:11:43.427372 30760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:11:43.510383 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 30625
I20260807 03:11:43.527393 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 30771
Aug 07 03:11:43 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:11:43.543260 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:43.528324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.17.31.2:33999)
0807 03:11:43.528840 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:43.528847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:43.528872 (+ 25us) server_negotiation.cc:407] Connection header received
0807 03:11:43.528930 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:43.528938 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:43.529153 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:43.529278 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:43.530562 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:43.531520 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:43.533061 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:43.533597 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:43.533689 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:43.537832 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:43.537843 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:43.537856 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:43.537885 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:43.540735 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:43.541311 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:43.541317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:43.541319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:43.541449 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:43.541750 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:43.541753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:43.541755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:43.542198 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:43.542519 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:43.542775 (+ 256us) server_negotiation.cc:299] Negotiation successful
0807 03:11:43.542961 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":270,"threads_started":1}
I20260807 03:11:43.544427 30909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:11:43.628626 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 30771
I20260807 03:11:43.650156 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 30920
Aug 07 03:11:43 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:11:43.666731 31065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:43.651138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47988 (local address 127.17.31.3:41715)
0807 03:11:43.651861 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:43.651868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:43.651896 (+ 28us) server_negotiation.cc:407] Connection header received
0807 03:11:43.651964 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:43.651970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:43.652215 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:43.652387 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:43.653805 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:43.655003 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:43.656436 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:43.656999 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:43.657137 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:43.661351 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:43.661361 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:43.661374 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:43.661408 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:43.663997 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:43.664640 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:43.664646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:43.664648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:43.664774 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:43.665148 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:43.665155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:43.665158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:43.665487 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:43.665807 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:43.666101 (+ 294us) server_negotiation.cc:299] Negotiation successful
0807 03:11:43.666382 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":364,"threads_started":1}
I20260807 03:11:43.668151 31051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:11:43.752168 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 30920
I20260807 03:11:43.769083 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 30506
I20260807 03:11:43.770396 30572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:11:43.875280 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 30506
I20260807 03:11:43.898435 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:36437
--webserver_interface=127.17.31.62
--webserver_port=41825
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:36437
--ranger_config_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.31.20:56181/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.31.20:56181/kms with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:11:44.223254 31068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:11:44.223524 31068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:11:44.223703 31068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:11:44.230741 31068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:11:44.230860 31068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:11:44.230979 31068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:11:44.231077 31068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:11:44.231163 31068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:11:44.240790 31068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.31.20:56181/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:36437
--ranger_config_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:36437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=41825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:11:44.241967 31068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:11:44.243351 31068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:11:44.254808 31075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:44.254743 31077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:44.255141 31073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:44.255916 31068 server_base.cc:1034] running on GCE node
I20260807 03:11:44.256573 31068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:11:44.257933 31068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:11:44.259124 31068 hybrid_clock.cc:648] HybridClock initialized: now 1786072304259077 us; error 48 us; skew 500 ppm
I20260807 03:11:44.259470 31068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:11:44 dist-test-slave-tm1g krb5kdc[30040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072304, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:44.262465 31068 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:11:44.263876 31068 webserver.cc:460] Webserver started at http://127.17.31.62:41825/ using document root <none> and password file <none>
I20260807 03:11:44.264503 31068 fs_manager.cc:359] Metadata directory not provided
I20260807 03:11:44.264652 31068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 03:11:44 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072304, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
I20260807 03:11:44.314158 31068 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 03:11:44.317009 31085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:44.318395 31068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:11:44.318596 31068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "7024138b70bf4671a96b0b8ac8c99668"
format_stamp: "Formatted at 2026-08-07 03:11:40 on dist-test-slave-tm1g"
server_key: "96a159bafd0922c52d478fd8d451019d"
server_key_iv: "ebdeaf100beb31482e8f3f547801e394"
server_key_version: "kuduclusterkey@0"
I20260807 03:11:44.319514 31068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:11:44.347826 31068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:11:44.351456 31068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:11:44.351737 31068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:11:44.370159 31068 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:36437
I20260807 03:11:44.370193 31139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:36437 every 8 connection(s)
I20260807 03:11:44.371737 31068 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:11:44.373266 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 31068
I20260807 03:11:44.374440 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.31.20:56181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:34729
--local_ip_for_outbound_sockets=127.17.31.1
--tserver_master_addrs=127.17.31.62:36437
--webserver_port=45023
--webserver_interface=127.17.31.1
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.31.20:56181/kms with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:11:44.381753 31140 sys_catalog.cc:263] Verifying existing consensus state
I20260807 03:11:44.386206 31140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668: Bootstrap starting.
I20260807 03:11:44.405005 31140 log.cc:824] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668: Log is configured to *not* fsync() on all Append() calls
I20260807 03:11:44.413761 31140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:11:44.414412 31140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668: Bootstrap complete.
I20260807 03:11:44.423491 31140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7024138b70bf4671a96b0b8ac8c99668" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36437 } }
I20260807 03:11:44.424398 31140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7024138b70bf4671a96b0b8ac8c99668, State: Initialized, Role: FOLLOWER
I20260807 03:11:44.424995 31140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7024138b70bf4671a96b0b8ac8c99668" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36437 } }
I20260807 03:11:44.425230 31140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:11:44.425385 31140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:11:44.425570 31140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:11:44.429352 31140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7024138b70bf4671a96b0b8ac8c99668" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36437 } }
I20260807 03:11:44.429839 31140 leader_election.cc:302] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7024138b70bf4671a96b0b8ac8c99668; no voters:
I20260807 03:11:44.430521 31140 leader_election.cc:288] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 03:11:44.430660 31144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:11:44.432838 31140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:11:44.437849 31144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [term 2 LEADER]: Becoming Leader. State: Replica: 7024138b70bf4671a96b0b8ac8c99668, State: Running, Role: LEADER
I20260807 03:11:44.438566 31144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7024138b70bf4671a96b0b8ac8c99668" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36437 } }
I20260807 03:11:44.441406 31145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7024138b70bf4671a96b0b8ac8c99668" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7024138b70bf4671a96b0b8ac8c99668" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36437 } } }
I20260807 03:11:44.441869 31145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [sys.catalog]: This master's current role is: LEADER
I20260807 03:11:44.442526 31146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7024138b70bf4671a96b0b8ac8c99668. Latest consensus state: current_term: 2 leader_uuid: "7024138b70bf4671a96b0b8ac8c99668" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7024138b70bf4671a96b0b8ac8c99668" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36437 } } }
I20260807 03:11:44.442796 31146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668 [sys.catalog]: This master's current role is: LEADER
I20260807 03:11:44.442831 31157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:11:44.454114 31157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:11:44.459306 31157 catalog_manager.cc:1259] Loaded cluster ID: 8678b75e63aa4b0d9a0d716fbbb6ca81
I20260807 03:11:44.459462 31157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:11:44.463388 31157 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:11:44.464844 31157 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668: Loaded TSK: 0
I20260807 03:11:44.466559 31157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:11:44.469421 31140 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260807 03:11:44.779596 31142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:11:44.780031 31142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:11:44.780246 31142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:11:44.791837 31142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:11:44.792038 31142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:11:44.792282 31142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:11:44.807711 31142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.31.20:56181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:34729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=45023
--tserver_master_addrs=127.17.31.62:36437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:11:44.809679 31142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:11:44.811573 31142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:11:44.844470 31179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:44.851846 31180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:44.859918 31142 server_base.cc:1034] running on GCE node
W20260807 03:11:44.860420 31183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:44.861317 31142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:11:44.873818 31142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:11:44.875377 31142 hybrid_clock.cc:648] HybridClock initialized: now 1786072304875302 us; error 79 us; skew 500 ppm
I20260807 03:11:44.875845 31142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:11:44 dist-test-slave-tm1g krb5kdc[30040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072304, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:44.880388 31142 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:11:44.882762 31142 webserver.cc:460] Webserver started at http://127.17.31.1:45023/ using document root <none> and password file <none>
I20260807 03:11:44.883733 31142 fs_manager.cc:359] Metadata directory not provided
I20260807 03:11:44.883991 31142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 03:11:44 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072304, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
I20260807 03:11:44.921074 31142 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.000s sys 0.007s
I20260807 03:11:44.924731 31190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:44.926595 31142 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260807 03:11:44.926844 31142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "a65cabb99b5a463c8c0c56692dc3c9d3"
format_stamp: "Formatted at 2026-08-07 03:11:41 on dist-test-slave-tm1g"
server_key: "dd9282e44f808d7fc6927a79636bc483"
server_key_iv: "f21184792277f1eb767498ad7d635cf3"
server_key_version: "kuduclusterkey@0"
I20260807 03:11:44.928220 31142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:11:44.954674 31142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:11:44.959621 31142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:11:44.959967 31142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:11:44.978756 31142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:11:44.986712 31142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:11:44.986884 31142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:44.987092 31142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:11:44.987229 31142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:45.086136 31142 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:34729
I20260807 03:11:45.086182 31308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:34729 every 8 connection(s)
I20260807 03:11:45.087628 31142 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:11:45.092969 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 31142
I20260807 03:11:45.093787 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.31.20:56181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:33999
--local_ip_for_outbound_sockets=127.17.31.2
--tserver_master_addrs=127.17.31.62:36437
--webserver_port=36269
--webserver_interface=127.17.31.2
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.31.20:56181/kms with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 03:11:45 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072304, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:11:45.132438 31311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:45.091836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:36437 (local address 127.17.31.1:34803)
0807 03:11:45.093382 (+ 1546us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:45.093448 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:45.096189 (+ 2741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:45.109981 (+ 13792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:45.109993 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:45.110631 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:45.111433 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:45.111459 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:45.113880 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:45.113889 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:45.116226 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:45.116245 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:45.117763 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:45.117771 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:45.118055 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:45.118895 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:45.118936 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:45.121963 (+ 3027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:45.127183 (+ 5220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:45.127193 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:45.127208 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:45.127830 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:45.128417 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:45.128422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:45.128425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:45.128694 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:45.129762 (+ 1068us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:45.129771 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:45.130151 (+ 380us) client_negotiation.cc:769] Sending connection context
0807 03:11:45.130495 (+ 344us) client_negotiation.cc:240] Negotiation successful
0807 03:11:45.131028 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1214,"thread_start_us":7369,"threads_started":1}
I20260807 03:11:45.135347 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:45.099381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:34803 (local address 127.17.31.62:36437)
0807 03:11:45.108306 (+ 8925us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:45.108321 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:45.108363 (+ 42us) server_negotiation.cc:407] Connection header received
0807 03:11:45.108770 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:45.108794 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:45.109338 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:45.109747 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:45.111685 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:45.113693 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:45.116409 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:45.117477 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:45.118495 (+ 1018us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:45.122292 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:45.122324 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:45.122346 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:45.122401 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:45.127005 (+ 4604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:45.128011 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:45.128030 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:45.128041 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:45.128264 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:45.128844 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:45.128850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:45.128856 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:45.129598 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:45.129824 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:45.132909 (+ 3085us) server_negotiation.cc:299] Negotiation successful
0807 03:11:45.133295 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8684,"thread_start_us":8455,"threads_started":1}
I20260807 03:11:45.138796 31309 heartbeater.cc:347] Connected to a master server at 127.17.31.62:36437
I20260807 03:11:45.139112 31309 heartbeater.cc:464] Registering TS with master...
I20260807 03:11:45.139873 31309 heartbeater.cc:511] Master 127.17.31.62:36437 requested a full tablet report, sending...
W20260807 03:11:45.142762 31309 heartbeater.cc:644] Failed to heartbeat to 127.17.31.62:36437 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 03:11:45.143014 31309 heartbeater.cc:464] Registering TS with master...
I20260807 03:11:45.143257 31309 heartbeater.cc:511] Master 127.17.31.62:36437 requested a full tablet report, sending...
I20260807 03:11:45.144543 31309 heartbeater.cc:464] Registering TS with master...
I20260807 03:11:45.144807 31309 heartbeater.cc:511] Master 127.17.31.62:36437 requested a full tablet report, sending...
W20260807 03:11:45.145890 31309 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 03:11:45.506080 31313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:11:45.506403 31313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:11:45.506662 31313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:11:45.516177 31313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:11:45.516299 31313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:11:45.516530 31313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:11:45.530375 31313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.31.20:56181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:33999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=36269
--tserver_master_addrs=127.17.31.62:36437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:11:45.531971 31313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:11:45.533612 31313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:11:45.549852 31327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:45.551400 31328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:45.551826 31330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:45.552626 31313 server_base.cc:1034] running on GCE node
I20260807 03:11:45.553203 31313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:11:45.554184 31313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:11:45.555469 31313 hybrid_clock.cc:648] HybridClock initialized: now 1786072305555417 us; error 44 us; skew 500 ppm
I20260807 03:11:45.555847 31313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:11:45 dist-test-slave-tm1g krb5kdc[30040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072305, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:45.562790 31313 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:11:45.585210 31313 webserver.cc:460] Webserver started at http://127.17.31.2:36269/ using document root <none> and password file <none>
I20260807 03:11:45.586233 31313 fs_manager.cc:359] Metadata directory not provided
I20260807 03:11:45.586447 31313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 03:11:45 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072305, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
I20260807 03:11:45.644555 31313 fs_manager.cc:709] Time spent opening directory manager: real 0.031s user 0.014s sys 0.015s
I20260807 03:11:45.648475 31340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:45.650235 31313 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260807 03:11:45.650449 31313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "02f6dcef6a064f56b054922e20a7de72"
format_stamp: "Formatted at 2026-08-07 03:11:42 on dist-test-slave-tm1g"
server_key: "590cea6c5c34726d7247da3c31fb8dd2"
server_key_iv: "98d961126ad07267ceeb9a508312a010"
server_key_version: "kuduclusterkey@0"
I20260807 03:11:45.651837 31313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 07 03:11:45 dist-test-slave-tm1g krb5kdc[30040](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786072305, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:45.715648 31313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:11:45.720724 31313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:11:45.721071 31313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:11:45.722358 31313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:11:45.724323 31313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:11:45.724478 31313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:45.724643 31313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:11:45.724761 31313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:45.786015 31313 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:33999
I20260807 03:11:45.786051 31458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:33999 every 8 connection(s)
I20260807 03:11:45.787942 31313 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:11:45.790659 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 31313
I20260807 03:11:45.791471 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.31.20:56181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:41715
--local_ip_for_outbound_sockets=127.17.31.3
--tserver_master_addrs=127.17.31.62:36437
--webserver_port=33235
--webserver_interface=127.17.31.3
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.31.20:56181/kms with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 03:11:45 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072305, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:11:45.818846 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:45.792020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:53179 (local address 127.17.31.62:36437)
0807 03:11:45.792578 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:45.792586 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:45.795804 (+ 3218us) server_negotiation.cc:407] Connection header received
0807 03:11:45.799694 (+ 3890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:45.799701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:45.799800 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:45.799935 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:45.801554 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:45.802483 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:45.805044 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:45.805858 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:45.805981 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:45.810760 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:45.810768 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:45.810775 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:45.810819 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:45.814584 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:45.815586 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:45.815594 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:45.815600 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:45.815760 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:45.816377 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:45.816386 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:45.816392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:45.816812 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:45.816974 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:45.818056 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0807 03:11:45.818353 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":249,"threads_started":1}
I20260807 03:11:45.821348 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:45.791403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:36437 (local address 127.17.31.2:53179)
0807 03:11:45.795630 (+ 4227us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:45.795683 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:45.799375 (+ 3692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:45.800177 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:45.800189 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:45.800815 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:45.801382 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:45.801398 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:45.802674 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:45.802682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:45.804875 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:45.804892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:45.806376 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:45.806387 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:45.806740 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:45.807538 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:45.807572 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:45.810529 (+ 2957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:45.814762 (+ 4233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:45.814773 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:45.814792 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:45.815432 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:45.815938 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:45.815944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:45.815951 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:45.816232 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:45.817100 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:45.817110 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:45.817546 (+ 436us) client_negotiation.cc:769] Sending connection context
0807 03:11:45.819179 (+ 1633us) client_negotiation.cc:240] Negotiation successful
0807 03:11:45.819785 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3884,"thread_start_us":3658,"threads_started":1}
I20260807 03:11:45.823390 31459 heartbeater.cc:347] Connected to a master server at 127.17.31.62:36437
I20260807 03:11:45.823719 31459 heartbeater.cc:464] Registering TS with master...
I20260807 03:11:45.824671 31459 heartbeater.cc:511] Master 127.17.31.62:36437 requested a full tablet report, sending...
I20260807 03:11:45.826807 31103 ts_manager.cc:194] Registered new tserver with Master: 02f6dcef6a064f56b054922e20a7de72 (127.17.31.2:33999)
I20260807 03:11:45.828938 31103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:53179
I20260807 03:11:46.146025 31309 heartbeater.cc:464] Registering TS with master...
I20260807 03:11:46.146513 31309 heartbeater.cc:511] Master 127.17.31.62:36437 requested a full tablet report, sending...
I20260807 03:11:46.148171 31103 ts_manager.cc:194] Registered new tserver with Master: a65cabb99b5a463c8c0c56692dc3c9d3 (127.17.31.1:34729)
I20260807 03:11:46.149363 31103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:34803
Aug 07 03:11:46 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072305, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
W20260807 03:11:46.250902 31464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:11:46.251380 31464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:11:46.251556 31464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:11:46.262549 31464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:11:46.262753 31464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:11:46.262966 31464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:11:46.276727 31464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.31.20:56181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:41715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=33235
--tserver_master_addrs=127.17.31.62:36437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:11:46.278671 31464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:11:46.280378 31464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:11:46.298439 31469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:46.300661 31470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:46.309187 31472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:46.309657 31464 server_base.cc:1034] running on GCE node
I20260807 03:11:46.310382 31464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:11:46.311414 31464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:11:46.312630 31464 hybrid_clock.cc:648] HybridClock initialized: now 1786072306312588 us; error 36 us; skew 500 ppm
I20260807 03:11:46.312949 31464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:11:46 dist-test-slave-tm1g krb5kdc[30040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072306, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:46.316586 31464 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:11:46.318594 31464 webserver.cc:460] Webserver started at http://127.17.31.3:33235/ using document root <none> and password file <none>
I20260807 03:11:46.319525 31464 fs_manager.cc:359] Metadata directory not provided
I20260807 03:11:46.319675 31464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 03:11:46 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072306, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
I20260807 03:11:46.355684 31464 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.005s sys 0.009s
I20260807 03:11:46.359642 31481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:46.361464 31464 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260807 03:11:46.361639 31464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "7f05876e7f6b4af98735aaa04d253392"
format_stamp: "Formatted at 2026-08-07 03:11:43 on dist-test-slave-tm1g"
server_key: "f24186349986976c38cbb297fa883870"
server_key_iv: "eaaafc26ca83ee03db6beb3d2cf46778"
server_key_version: "kuduclusterkey@0"
I20260807 03:11:46.363025 31464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:11:46.396454 31464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:11:46.401662 31464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:11:46.401993 31464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:11:46.403223 31464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:11:46.405040 31464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:11:46.405179 31464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260807 03:11:46.405311 31464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:11:46.405411 31464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:46.467079 31464 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:41715
I20260807 03:11:46.467173 31599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:41715 every 8 connection(s)
I20260807 03:11:46.469110 31464 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:11:46.478443 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 31464
Aug 07 03:11:46 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072306, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
Aug 07 03:11:46 dist-test-slave-tm1g krb5kdc[30040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072305, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for HTTP/127.17.31.20@KRBTEST.COM
I20260807 03:11:46.508535 31602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:46.473381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:36437 (local address 127.17.31.3:47639)
0807 03:11:46.478899 (+ 5518us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:46.478959 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:46.485163 (+ 6204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:46.486240 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:46.486251 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:46.486841 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:46.487667 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:46.487693 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.489286 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.489291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:46.492221 (+ 2930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:46.492236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.493593 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.493600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:46.493876 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:46.494719 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:46.494748 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:46.497418 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:46.502030 (+ 4612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:46.502039 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:46.502056 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:46.502657 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:46.504816 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:46.504821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:46.504824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:46.505063 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:46.505825 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:46.505833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:46.506247 (+ 414us) client_negotiation.cc:769] Sending connection context
0807 03:11:46.506601 (+ 354us) client_negotiation.cc:240] Negotiation successful
0807 03:11:46.507139 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5198,"thread_start_us":4979,"threads_started":1}
I20260807 03:11:46.509828 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:46.473934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:47639 (local address 127.17.31.62:36437)
0807 03:11:46.482686 (+ 8752us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:46.482690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:46.482708 (+ 18us) server_negotiation.cc:407] Connection header received
0807 03:11:46.485501 (+ 2793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:46.485507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:46.485597 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:46.485714 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:46.487910 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.489087 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.492419 (+ 3332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.493132 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.493228 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:46.497712 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:46.497718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:46.497722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:46.497791 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:46.501179 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:46.504541 (+ 3362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:46.504546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:46.504550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:46.504682 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:46.505214 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:46.505219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:46.505223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:46.505670 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:46.505827 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:46.509229 (+ 3402us) server_negotiation.cc:299] Negotiation successful
0807 03:11:46.509449 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8642,"thread_start_us":8546,"threads_started":1}
I20260807 03:11:46.511108 31600 heartbeater.cc:347] Connected to a master server at 127.17.31.62:36437
I20260807 03:11:46.511416 31600 heartbeater.cc:464] Registering TS with master...
I20260807 03:11:46.512247 31600 heartbeater.cc:511] Master 127.17.31.62:36437 requested a full tablet report, sending...
I20260807 03:11:46.513944 31103 ts_manager.cc:194] Registered new tserver with Master: 7f05876e7f6b4af98735aaa04d253392 (127.17.31.3:41715)
I20260807 03:11:46.515085 31103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:47639
I20260807 03:11:46.516248 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:46.482663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60444 (local address 127.17.31.62:36437)
0807 03:11:46.483159 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:46.483166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:46.483185 (+ 19us) server_negotiation.cc:407] Connection header received
0807 03:11:46.483244 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:46.483249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:46.483360 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:46.485598 (+ 2238us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:46.485747 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.486906 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.489068 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.489730 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.490477 (+ 747us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:46.497266 (+ 6789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:46.497273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:46.497278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:46.497316 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:46.504361 (+ 7045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:46.505226 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:46.505231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:46.505235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:46.505433 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:46.513174 (+ 7741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:46.513179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:46.513183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:46.513550 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:46.513805 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:46.515647 (+ 1842us) server_negotiation.cc:299] Negotiation successful
0807 03:11:46.515870 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":253,"threads_started":1}
I20260807 03:11:46.520787 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:11:46.605497 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:46.569652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.17.31.62:36437)
0807 03:11:46.569880 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:46.569887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:46.582938 (+ 13051us) server_negotiation.cc:407] Connection header received
0807 03:11:46.583288 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:46.583295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:46.583396 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:46.583469 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:46.585825 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.587030 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.592455 (+ 5425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.593120 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.593219 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:46.598528 (+ 5309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:46.598535 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:46.598540 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:46.598582 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:46.602194 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:46.603079 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:46.603085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:46.603089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:46.603234 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:46.603709 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:46.603715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:46.603718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:46.604082 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:46.604303 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:46.604902 (+ 599us) server_negotiation.cc:299] Negotiation successful
0807 03:11:46.605140 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 03:11:46.612660 31103 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:11:46.624846 31103 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:11:46.695565 31618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:46.666412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:41715 (local address 127.0.0.1:47998)
0807 03:11:46.670202 (+ 3790us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:46.670224 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:46.673004 (+ 2780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:46.680616 (+ 7612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:46.680624 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:46.680675 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:11:46.681085 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:46.681099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.683225 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.683231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:46.685687 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:46.685703 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.694438 (+ 8735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.694445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:46.694668 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:46.694725 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 03:11:46.694891 (+ 166us) client_negotiation.cc:240] Negotiation successful
0807 03:11:46.695084 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1851,"spinlock_wait_cycles":8832,"thread_start_us":3955,"threads_started":1}
I20260807 03:11:46.701272 31619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:46.667500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.17.31.3:41715)
0807 03:11:46.677917 (+ 10417us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:46.677926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:46.677961 (+ 35us) server_negotiation.cc:407] Connection header received
0807 03:11:46.678070 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:46.678078 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:46.678384 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:46.678627 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:11:46.681296 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.683057 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.692441 (+ 9384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.693898 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.694028 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:46.694251 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:46.700732 (+ 6481us) server_negotiation.cc:299] Negotiation successful
0807 03:11:46.700920 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10235,"thread_start_us":12192,"threads_started":1}
I20260807 03:11:46.698074 31621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:46.669053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:34729 (local address 127.0.0.1:52904)
0807 03:11:46.680545 (+ 11492us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:46.680563 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:46.680691 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:46.682668 (+ 1977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:46.682675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:46.682705 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:11:46.683080 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:46.683096 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.688608 (+ 5512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.688615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:46.690722 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:46.690734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.697223 (+ 6489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.697229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:46.697424 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:46.697463 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 03:11:46.697545 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 03:11:46.697676 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11353,"spinlock_wait_cycles":9088,"thread_start_us":11214,"threads_started":1}
I20260807 03:11:46.702862 31617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:46.666660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:33999 (local address 127.0.0.1:45464)
0807 03:11:46.669447 (+ 2787us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:46.669611 (+ 164us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:46.682531 (+ 12920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:46.685797 (+ 3266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:46.685818 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:46.685845 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:11:46.686269 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:46.686283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.693156 (+ 6873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.693163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:46.695283 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:46.695300 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.702052 (+ 6752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.702058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:46.702224 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:46.702254 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 03:11:46.702359 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 03:11:46.702476 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2089,"mutex_wait_us":1,"thread_start_us":868,"threads_started":1}
I20260807 03:11:46.704104 31620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:46.668090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.17.31.2:33999)
0807 03:11:46.682421 (+ 14331us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:46.682428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:46.682457 (+ 29us) server_negotiation.cc:407] Connection header received
0807 03:11:46.683372 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:46.683382 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:46.683611 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:46.686472 (+ 2861us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:11:46.686699 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.688453 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.695501 (+ 7048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.697011 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.703343 (+ 6332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:46.703524 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:46.703592 (+ 68us) server_negotiation.cc:299] Negotiation successful
0807 03:11:46.703758 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14150,"thread_start_us":14188,"threads_started":1}
I20260807 03:11:46.704891 31622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:46.669733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.17.31.1:34729)
0807 03:11:46.681239 (+ 11506us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:46.681247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:46.681275 (+ 28us) server_negotiation.cc:407] Connection header received
0807 03:11:46.681372 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:46.681379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:46.681627 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:46.681840 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:11:46.683866 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.685614 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.690920 (+ 5306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:46.692603 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:46.692739 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:46.693050 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:46.704383 (+ 11333us) server_negotiation.cc:299] Negotiation successful
0807 03:11:46.704565 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11342,"thread_start_us":11230,"threads_started":1}
I20260807 03:11:46.713349 31243 tablet_service.cc:1467] Processing CreateTablet for tablet 1258c14015ec4d6885a472c1ae7272b9 (DEFAULT_TABLE table=test-table [id=479a7fb393f64df7823b39a797d0f4cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:11:46.710037 31392 tablet_service.cc:1467] Processing CreateTablet for tablet 1258c14015ec4d6885a472c1ae7272b9 (DEFAULT_TABLE table=test-table [id=479a7fb393f64df7823b39a797d0f4cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:11:46.720214 31392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1258c14015ec4d6885a472c1ae7272b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:11:46.727164 31243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1258c14015ec4d6885a472c1ae7272b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:11:46.705971 31532 tablet_service.cc:1467] Processing CreateTablet for tablet 1258c14015ec4d6885a472c1ae7272b9 (DEFAULT_TABLE table=test-table [id=479a7fb393f64df7823b39a797d0f4cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:11:46.729732 31532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1258c14015ec4d6885a472c1ae7272b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:11:46.763669 31623 tablet_bootstrap.cc:492] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72: Bootstrap starting.
I20260807 03:11:46.770973 31624 tablet_bootstrap.cc:492] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392: Bootstrap starting.
I20260807 03:11:46.772668 31625 tablet_bootstrap.cc:492] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3: Bootstrap starting.
I20260807 03:11:46.774508 31624 tablet_bootstrap.cc:654] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392: Neither blocks nor log segments found. Creating new log.
I20260807 03:11:46.775821 31624 log.cc:824] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392: Log is configured to *not* fsync() on all Append() calls
I20260807 03:11:46.776214 31625 tablet_bootstrap.cc:654] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3: Neither blocks nor log segments found. Creating new log.
I20260807 03:11:46.777447 31625 log.cc:824] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3: Log is configured to *not* fsync() on all Append() calls
I20260807 03:11:46.784884 31624 tablet_bootstrap.cc:492] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392: No bootstrap required, opened a new log
I20260807 03:11:46.785185 31624 ts_tablet_manager.cc:1397] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.013s
I20260807 03:11:46.774255 31623 tablet_bootstrap.cc:654] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72: Neither blocks nor log segments found. Creating new log.
I20260807 03:11:46.791813 31623 log.cc:824] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72: Log is configured to *not* fsync() on all Append() calls
I20260807 03:11:46.793004 31625 tablet_bootstrap.cc:492] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3: No bootstrap required, opened a new log
I20260807 03:11:46.793262 31625 ts_tablet_manager.cc:1397] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.018s
I20260807 03:11:46.795886 31624 raft_consensus.cc:348] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f05876e7f6b4af98735aaa04d253392" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 41715 } } peers { permanent_uuid: "02f6dcef6a064f56b054922e20a7de72" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 33999 } } peers { permanent_uuid: "a65cabb99b5a463c8c0c56692dc3c9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 34729 } }
I20260807 03:11:46.796388 31624 raft_consensus.cc:374] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:11:46.796523 31624 raft_consensus.cc:729] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f05876e7f6b4af98735aaa04d253392, State: Initialized, Role: FOLLOWER
I20260807 03:11:46.797277 31624 consensus_queue.cc:260] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f05876e7f6b4af98735aaa04d253392" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 41715 } } peers { permanent_uuid: "02f6dcef6a064f56b054922e20a7de72" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 33999 } } peers { permanent_uuid: "a65cabb99b5a463c8c0c56692dc3c9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 34729 } }
I20260807 03:11:46.804095 31624 ts_tablet_manager.cc:1428] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392: Time spent starting tablet: real 0.019s user 0.010s sys 0.004s
I20260807 03:11:46.804715 31600 heartbeater.cc:503] Master 127.17.31.62:36437 was elected leader, sending a full tablet report...
I20260807 03:11:46.804828 31623 tablet_bootstrap.cc:492] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72: No bootstrap required, opened a new log
I20260807 03:11:46.805073 31623 ts_tablet_manager.cc:1397] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.006s
I20260807 03:11:46.812922 31625 raft_consensus.cc:348] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f05876e7f6b4af98735aaa04d253392" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 41715 } } peers { permanent_uuid: "02f6dcef6a064f56b054922e20a7de72" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 33999 } } peers { permanent_uuid: "a65cabb99b5a463c8c0c56692dc3c9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 34729 } }
I20260807 03:11:46.813448 31625 raft_consensus.cc:374] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:11:46.813596 31625 raft_consensus.cc:729] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a65cabb99b5a463c8c0c56692dc3c9d3, State: Initialized, Role: FOLLOWER
I20260807 03:11:46.814361 31625 consensus_queue.cc:260] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f05876e7f6b4af98735aaa04d253392" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 41715 } } peers { permanent_uuid: "02f6dcef6a064f56b054922e20a7de72" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 33999 } } peers { permanent_uuid: "a65cabb99b5a463c8c0c56692dc3c9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 34729 } }
I20260807 03:11:46.814972 31623 raft_consensus.cc:348] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f05876e7f6b4af98735aaa04d253392" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 41715 } } peers { permanent_uuid: "02f6dcef6a064f56b054922e20a7de72" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 33999 } } peers { permanent_uuid: "a65cabb99b5a463c8c0c56692dc3c9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 34729 } }
I20260807 03:11:46.815454 31623 raft_consensus.cc:374] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:11:46.815582 31623 raft_consensus.cc:729] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f6dcef6a064f56b054922e20a7de72, State: Initialized, Role: FOLLOWER
I20260807 03:11:46.816251 31623 consensus_queue.cc:260] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f05876e7f6b4af98735aaa04d253392" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 41715 } } peers { permanent_uuid: "02f6dcef6a064f56b054922e20a7de72" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 33999 } } peers { permanent_uuid: "a65cabb99b5a463c8c0c56692dc3c9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 34729 } }
I20260807 03:11:46.816833 31309 heartbeater.cc:503] Master 127.17.31.62:36437 was elected leader, sending a full tablet report...
I20260807 03:11:46.817404 31459 heartbeater.cc:503] Master 127.17.31.62:36437 was elected leader, sending a full tablet report...
I20260807 03:11:46.817510 31625 ts_tablet_manager.cc:1428] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3: Time spent starting tablet: real 0.024s user 0.011s sys 0.004s
I20260807 03:11:46.819511 31623 ts_tablet_manager.cc:1428] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260807 03:11:46.850687 31310 tablet.cc:2369] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:11:46.977880 31601 tablet.cc:2369] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:11:47.043210 31460 tablet.cc:2369] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:11:47.066138 31629 raft_consensus.cc:482] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:11:47.066468 31629 raft_consensus.cc:504] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f05876e7f6b4af98735aaa04d253392" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 41715 } } peers { permanent_uuid: "02f6dcef6a064f56b054922e20a7de72" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 33999 } } peers { permanent_uuid: "a65cabb99b5a463c8c0c56692dc3c9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 34729 } }
I20260807 03:11:47.068456 31629 leader_election.cc:288] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02f6dcef6a064f56b054922e20a7de72 (127.17.31.2:33999), a65cabb99b5a463c8c0c56692dc3c9d3 (127.17.31.1:34729)
I20260807 03:11:47.071990 31631 raft_consensus.cc:482] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:11:47.072293 31631 raft_consensus.cc:504] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f05876e7f6b4af98735aaa04d253392" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 41715 } } peers { permanent_uuid: "02f6dcef6a064f56b054922e20a7de72" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 33999 } } peers { permanent_uuid: "a65cabb99b5a463c8c0c56692dc3c9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 34729 } }
I20260807 03:11:47.074168 31631 leader_election.cc:288] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f05876e7f6b4af98735aaa04d253392 (127.17.31.3:41715), a65cabb99b5a463c8c0c56692dc3c9d3 (127.17.31.1:34729)
I20260807 03:11:47.078321 31620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:47.068957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:38693 (local address 127.17.31.2:33999)
0807 03:11:47.069159 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:47.069165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:47.069916 (+ 751us) server_negotiation.cc:407] Connection header received
0807 03:11:47.070234 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:47.070240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:47.070340 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:47.070435 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:11:47.071078 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.072194 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.075137 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.076743 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.076866 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:47.076982 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:47.077647 (+ 665us) server_negotiation.cc:299] Negotiation successful
0807 03:11:47.077860 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 03:11:47.079339 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:47.069170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:33999 (local address 127.17.31.3:38693)
0807 03:11:47.069792 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:47.069814 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:47.070033 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:47.070446 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:47.070451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:47.070478 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:11:47.070911 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:47.070923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.072395 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.072403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:47.074902 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:47.074927 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.077132 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.077140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:47.077396 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:47.077457 (+ 61us) client_negotiation.cc:769] Sending connection context
0807 03:11:47.078560 (+ 1103us) client_negotiation.cc:240] Negotiation successful
0807 03:11:47.078766 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":54,"thread_start_us":327,"threads_started":1}
I20260807 03:11:47.087883 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1258c14015ec4d6885a472c1ae7272b9" candidate_uuid: "7f05876e7f6b4af98735aaa04d253392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f6dcef6a064f56b054922e20a7de72" is_pre_election: true
I20260807 03:11:47.088733 31414 raft_consensus.cc:2454] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f05876e7f6b4af98735aaa04d253392 in term 0.
I20260807 03:11:47.089869 31484 leader_election.cc:302] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f6dcef6a064f56b054922e20a7de72, 7f05876e7f6b4af98735aaa04d253392; no voters:
I20260807 03:11:47.090392 31629 raft_consensus.cc:2781] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:11:47.090538 31629 raft_consensus.cc:482] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:11:47.090631 31629 raft_consensus.cc:3037] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:11:47.093253 31622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:47.069055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:42405 (local address 127.17.31.1:34729)
0807 03:11:47.069252 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:47.069258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:47.070175 (+ 917us) server_negotiation.cc:407] Connection header received
0807 03:11:47.070434 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:47.070439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:47.070532 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:47.070644 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:11:47.071388 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.086525 (+ 15137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.089814 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.091518 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.091651 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:47.091787 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:47.092828 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0807 03:11:47.092958 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 03:11:47.096215 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:47.069435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:34729 (local address 127.17.31.3:42405)
0807 03:11:47.070085 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:47.070103 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:47.070256 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:47.070665 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:47.070670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:47.070700 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:11:47.071202 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:47.071214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.087266 (+ 16052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.087272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:47.089617 (+ 2345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:47.089635 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.092405 (+ 2770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.092417 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:47.092654 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:47.092707 (+ 53us) client_negotiation.cc:769] Sending connection context
0807 03:11:47.092802 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 03:11:47.092947 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":3,"thread_start_us":368,"threads_started":1}
I20260807 03:11:47.098832 31629 raft_consensus.cc:504] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f05876e7f6b4af98735aaa04d253392" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 41715 } } peers { permanent_uuid: "02f6dcef6a064f56b054922e20a7de72" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 33999 } } peers { permanent_uuid: "a65cabb99b5a463c8c0c56692dc3c9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 34729 } }
I20260807 03:11:47.100149 31263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1258c14015ec4d6885a472c1ae7272b9" candidate_uuid: "7f05876e7f6b4af98735aaa04d253392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a65cabb99b5a463c8c0c56692dc3c9d3" is_pre_election: true
I20260807 03:11:47.100678 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1258c14015ec4d6885a472c1ae7272b9" candidate_uuid: "7f05876e7f6b4af98735aaa04d253392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f6dcef6a064f56b054922e20a7de72"
I20260807 03:11:47.100745 31263 raft_consensus.cc:2454] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f05876e7f6b4af98735aaa04d253392 in term 0.
I20260807 03:11:47.101068 31414 raft_consensus.cc:3037] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:11:47.101954 31629 leader_election.cc:288] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [CANDIDATE]: Term 1 election: Requested vote from peers 02f6dcef6a064f56b054922e20a7de72 (127.17.31.2:33999), a65cabb99b5a463c8c0c56692dc3c9d3 (127.17.31.1:34729)
I20260807 03:11:47.102439 31263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1258c14015ec4d6885a472c1ae7272b9" candidate_uuid: "7f05876e7f6b4af98735aaa04d253392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a65cabb99b5a463c8c0c56692dc3c9d3"
I20260807 03:11:47.102730 31263 raft_consensus.cc:3037] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:11:47.106675 31414 raft_consensus.cc:2454] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f05876e7f6b4af98735aaa04d253392 in term 1.
I20260807 03:11:47.107028 31263 raft_consensus.cc:2454] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f05876e7f6b4af98735aaa04d253392 in term 1.
I20260807 03:11:47.107234 31484 leader_election.cc:302] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f6dcef6a064f56b054922e20a7de72, 7f05876e7f6b4af98735aaa04d253392; no voters:
I20260807 03:11:47.107587 31629 raft_consensus.cc:2781] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:11:47.109625 31622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:47.075174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:47689 (local address 127.17.31.1:34729)
0807 03:11:47.093518 (+ 18344us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:47.093524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:47.093552 (+ 28us) server_negotiation.cc:407] Connection header received
0807 03:11:47.098285 (+ 4733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:47.098290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:47.098411 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:47.098481 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:11:47.100650 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.102529 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.106180 (+ 3651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.108038 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.108166 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:47.108331 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:47.109104 (+ 773us) server_negotiation.cc:299] Negotiation successful
0807 03:11:47.109243 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18224,"threads_started":1}
I20260807 03:11:47.110879 31644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:47.074776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:34729 (local address 127.17.31.2:47689)
0807 03:11:47.087064 (+ 12288us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:47.087081 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:47.094386 (+ 7305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:47.099199 (+ 4813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:47.099204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:47.099231 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:11:47.099968 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:47.099983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.102761 (+ 2778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.102769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:47.105879 (+ 3110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:47.105893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.108652 (+ 2759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.108662 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:47.108881 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:47.108921 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 03:11:47.109880 (+ 959us) client_negotiation.cc:240] Negotiation successful
0807 03:11:47.110023 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12143,"threads_started":1}
I20260807 03:11:47.118450 31619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:47.086045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:48299 (local address 127.17.31.3:41715)
0807 03:11:47.086271 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:47.086277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:47.092714 (+ 6437us) server_negotiation.cc:407] Connection header received
0807 03:11:47.098765 (+ 6051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:47.098772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:47.098871 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:47.098937 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:11:47.108859 (+ 9922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.110214 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.114797 (+ 4583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.116779 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.116917 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:47.117061 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:47.117839 (+ 778us) server_negotiation.cc:299] Negotiation successful
0807 03:11:47.118042 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 03:11:47.120018 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:47.086023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:41715 (local address 127.17.31.2:48299)
0807 03:11:47.092622 (+ 6599us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:47.092641 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:47.094585 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:47.100149 (+ 5564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:47.100154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:47.100194 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:11:47.108672 (+ 8478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:47.108683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.110488 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.110493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:47.114523 (+ 4030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:47.114536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.117309 (+ 2773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.117318 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:47.117604 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:47.117657 (+ 53us) client_negotiation.cc:769] Sending connection context
0807 03:11:47.118891 (+ 1234us) client_negotiation.cc:240] Negotiation successful
0807 03:11:47.119126 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6428,"spinlock_wait_cycles":1570688,"thread_start_us":671,"threads_started":1}
I20260807 03:11:47.121670 31552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1258c14015ec4d6885a472c1ae7272b9" candidate_uuid: "02f6dcef6a064f56b054922e20a7de72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f05876e7f6b4af98735aaa04d253392" is_pre_election: true
I20260807 03:11:47.124620 31629 raft_consensus.cc:686] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [term 1 LEADER]: Becoming Leader. State: Replica: 7f05876e7f6b4af98735aaa04d253392, State: Running, Role: LEADER
I20260807 03:11:47.124711 31263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1258c14015ec4d6885a472c1ae7272b9" candidate_uuid: "02f6dcef6a064f56b054922e20a7de72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a65cabb99b5a463c8c0c56692dc3c9d3" is_pre_election: true
I20260807 03:11:47.125216 31263 raft_consensus.cc:2379] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02f6dcef6a064f56b054922e20a7de72 in current term 1: Already voted for candidate 7f05876e7f6b4af98735aaa04d253392 in this term.
I20260807 03:11:47.125486 31629 consensus_queue.cc:237] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f05876e7f6b4af98735aaa04d253392" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 41715 } } peers { permanent_uuid: "02f6dcef6a064f56b054922e20a7de72" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 33999 } } peers { permanent_uuid: "a65cabb99b5a463c8c0c56692dc3c9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 34729 } }
I20260807 03:11:47.131769 31343 leader_election.cc:302] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02f6dcef6a064f56b054922e20a7de72; no voters: 7f05876e7f6b4af98735aaa04d253392, a65cabb99b5a463c8c0c56692dc3c9d3
I20260807 03:11:47.132217 31631 raft_consensus.cc:2726] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:11:47.132551 31103 catalog_manager.cc:5495] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f05876e7f6b4af98735aaa04d253392 (127.17.31.3). New cstate: current_term: 1 leader_uuid: "7f05876e7f6b4af98735aaa04d253392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f05876e7f6b4af98735aaa04d253392" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 41715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02f6dcef6a064f56b054922e20a7de72" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 33999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a65cabb99b5a463c8c0c56692dc3c9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 34729 } health_report { overall_health: UNKNOWN } } }
I20260807 03:11:47.157943 31619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:47.148872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48012 (local address 127.17.31.3:41715)
0807 03:11:47.149104 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:47.149114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:47.149857 (+ 743us) server_negotiation.cc:407] Connection header received
0807 03:11:47.150111 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:47.150116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:47.150219 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:47.150428 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:11:47.151190 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.152249 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.155062 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.155696 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.155799 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:47.156504 (+ 705us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:11:47.157124 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:11:47.157271 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:47.157365 (+ 94us) server_negotiation.cc:299] Negotiation successful
0807 03:11:47.157432 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 03:11:47.168224 31414 raft_consensus.cc:1261] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72 [term 1 FOLLOWER]: Refusing update from remote peer 7f05876e7f6b4af98735aaa04d253392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:11:47.168339 31263 raft_consensus.cc:1261] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3 [term 1 FOLLOWER]: Refusing update from remote peer 7f05876e7f6b4af98735aaa04d253392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:11:47.169317 31646 consensus_queue.cc:1035] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f6dcef6a064f56b054922e20a7de72" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:11:47.169811 31629 consensus_queue.cc:1035] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a65cabb99b5a463c8c0c56692dc3c9d3" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 34729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:11:47.183835 31652 mvcc.cc:205] Tried to move back new op lower bound from 7315752170148958208 to 7315752169992175616. Current Snapshot: MvccSnapshot[applied={T|T < 7315752170148958208}]
I20260807 03:11:47.190379 31651 mvcc.cc:205] Tried to move back new op lower bound from 7315752170148958208 to 7315752169992175616. Current Snapshot: MvccSnapshot[applied={T|T < 7315752170148958208}]
I20260807 03:11:47.205715 31103 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:11:47.206086 31103 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:11:47.209555 31103 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7024138b70bf4671a96b0b8ac8c99668: Sending DeleteTablet for 3 replicas of tablet 1258c14015ec4d6885a472c1ae7272b9
I20260807 03:11:47.210978 31532 tablet_service.cc:1514] Processing DeleteTablet for tablet 1258c14015ec4d6885a472c1ae7272b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:11:47 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:47998
I20260807 03:11:47.211228 31392 tablet_service.cc:1514] Processing DeleteTablet for tablet 1258c14015ec4d6885a472c1ae7272b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:11:47 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:45464
I20260807 03:11:47.211282 31243 tablet_service.cc:1514] Processing DeleteTablet for tablet 1258c14015ec4d6885a472c1ae7272b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:11:47 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:52904
I20260807 03:11:47.214461 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 31142
I20260807 03:11:47.225661 31664 tablet_replica.cc:330] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3: stopping tablet replica
I20260807 03:11:47.227195 31664 raft_consensus.cc:2229] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:11:47.230787 31663 tablet_replica.cc:330] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72: stopping tablet replica
I20260807 03:11:47.231045 31664 raft_consensus.cc:2258] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:11:47.232231 31662 tablet_replica.cc:330] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392: stopping tablet replica
I20260807 03:11:47.232350 31663 raft_consensus.cc:2229] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:11:47.233395 31663 raft_consensus.cc:2258] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:11:47.234736 31662 raft_consensus.cc:2229] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:11:47.236061 31662 raft_consensus.cc:2258] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:11:47.236584 31664 ts_tablet_manager.cc:1905] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:11:47.237870 31663 ts_tablet_manager.cc:1905] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:11:47.245713 31662 ts_tablet_manager.cc:1905] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:11:47.247313 31663 ts_tablet_manager.cc:1918] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:11:47.247475 31664 ts_tablet_manager.cc:1918] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:11:47.247610 31663 log.cc:1196] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/1258c14015ec4d6885a472c1ae7272b9
I20260807 03:11:47.247776 31664 log.cc:1196] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/1258c14015ec4d6885a472c1ae7272b9
I20260807 03:11:47.248623 31663 ts_tablet_manager.cc:1939] T 1258c14015ec4d6885a472c1ae7272b9 P 02f6dcef6a064f56b054922e20a7de72: Deleting consensus metadata
I20260807 03:11:47.248869 31664 ts_tablet_manager.cc:1939] T 1258c14015ec4d6885a472c1ae7272b9 P a65cabb99b5a463c8c0c56692dc3c9d3: Deleting consensus metadata
I20260807 03:11:47.249677 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:47.215412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52914 (local address 127.17.31.1:34729)
0807 03:11:47.216034 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:47.216043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:47.220047 (+ 4004us) server_negotiation.cc:407] Connection header received
0807 03:11:47.220254 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:47.220263 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:47.220382 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:47.220498 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:47.227012 (+ 6514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.230095 (+ 3083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.232571 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.233476 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.233630 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:47.239214 (+ 5584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:47.239234 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:47.239252 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:47.239294 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:47.245088 (+ 5794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:47.246317 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:47.246326 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:47.246330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:47.246632 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:47.247174 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:47.247182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:47.247188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:47.247713 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:47.248196 (+ 483us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:47.248972 (+ 776us) server_negotiation.cc:299] Negotiation successful
0807 03:11:47.249227 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372}
I20260807 03:11:47.250926 31089 catalog_manager.cc:4841] TS 02f6dcef6a064f56b054922e20a7de72 (127.17.31.2:33999): tablet 1258c14015ec4d6885a472c1ae7272b9 (table test-table [id=479a7fb393f64df7823b39a797d0f4cb]) successfully deleted
I20260807 03:11:47.251358 31090 catalog_manager.cc:4841] TS a65cabb99b5a463c8c0c56692dc3c9d3 (127.17.31.1:34729): tablet 1258c14015ec4d6885a472c1ae7272b9 (table test-table [id=479a7fb393f64df7823b39a797d0f4cb]) successfully deleted
I20260807 03:11:47.251924 31304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:11:47.257412 31662 ts_tablet_manager.cc:1918] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:11:47.257735 31662 log.cc:1196] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/1258c14015ec4d6885a472c1ae7272b9
I20260807 03:11:47.258463 31662 ts_tablet_manager.cc:1939] T 1258c14015ec4d6885a472c1ae7272b9 P 7f05876e7f6b4af98735aaa04d253392: Deleting consensus metadata
I20260807 03:11:47.260320 31087 catalog_manager.cc:4841] TS 7f05876e7f6b4af98735aaa04d253392 (127.17.31.3:41715): tablet 1258c14015ec4d6885a472c1ae7272b9 (table test-table [id=479a7fb393f64df7823b39a797d0f4cb]) successfully deleted
I20260807 03:11:47.353473 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 31142
I20260807 03:11:47.373337 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 31313
I20260807 03:11:47.392619 31620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:47.374438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.17.31.2:33999)
0807 03:11:47.374626 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:47.374632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:47.374840 (+ 208us) server_negotiation.cc:407] Connection header received
0807 03:11:47.375032 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:47.375038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:47.375121 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:47.375221 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:47.377787 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.378923 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.381293 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.381907 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.382022 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:47.385340 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:47.385356 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:47.385377 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:47.385416 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:47.389466 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:47.390341 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:47.390350 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:47.390354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:47.390523 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:47.390947 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:47.390952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:47.390956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:47.391359 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:47.391626 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:47.392028 (+ 402us) server_negotiation.cc:299] Negotiation successful
0807 03:11:47.392248 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 03:11:47.393195 31454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:11:47.497437 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 31313
I20260807 03:11:47.515954 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 31464
I20260807 03:11:47.532430 31619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:47.516805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48024 (local address 127.17.31.3:41715)
0807 03:11:47.517053 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:47.517061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:47.517115 (+ 54us) server_negotiation.cc:407] Connection header received
0807 03:11:47.517184 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:47.517189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:47.517313 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:47.517447 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:47.518869 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.520058 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.521397 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:47.522086 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:47.522240 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:47.525065 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:47.525077 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:47.525094 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:47.525150 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:47.529193 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:47.529878 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:47.529887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:47.529891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:47.530074 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:47.530505 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:47.530513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:47.530520 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:47.531202 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:47.531579 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:47.531781 (+ 202us) server_negotiation.cc:299] Negotiation successful
0807 03:11:47.532035 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260807 03:11:47.533403 31594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:11:47.650182 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 31464
I20260807 03:11:47.669929 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 31068
I20260807 03:11:47.671386 31135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:11:47.789279 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 31068
I20260807 03:11:47.816461 17532 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 03:11:48.151294 17532 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 03:11:48.151506 17532 mini_ranger.cc:66] Stopping Ranger...
I20260807 03:11:48.496354 17532 mini_ranger.cc:68] Stopped Ranger
2026-08-07 03:11:48.496 UTC [30073] LOG: received smart shutdown request
2026-08-07 03:11:48.499 UTC [30073] LOG: background worker "logical replication launcher" (PID 30083) exited with exit code 1
2026-08-07 03:11:48.500 UTC [30078] LOG: shutting down
2026-08-07 03:11:48.640 UTC [30073] LOG: database system is shut down
2026-08-07T03:11:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52642 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:11:48 dist-test-slave-tm1g krb5kdc[31678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:11:48 dist-test-slave-tm1g krb5kdc[31678](info): set up 2 sockets
Aug 07 03:11:48 dist-test-slave-tm1g krb5kdc[31678](info): commencing operation
krb5kdc: starting...
W20260807 03:11:49.894012 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:11:50 dist-test-slave-tm1g krb5kdc[31678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:11:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:11:50.125633 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:39973
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:39973,127.17.31.61:45185,127.17.31.60:39545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:11:50.448498 31691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:11:50.448760 31691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:11:50.448848 31691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:11:50.455681 31691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:11:50.455772 31691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:11:50.455818 31691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:11:50.455878 31691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:11:50.455922 31691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:11:50.465111 31691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:39973,127.17.31.61:45185,127.17.31.60:39545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:39973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:11:50.466396 31691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:11:50.467720 31691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:11:50.478958 31697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:50.479007 31700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:50.478917 31698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:50.479897 31691 server_base.cc:1034] running on GCE node
I20260807 03:11:50.480656 31691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:11:50.481956 31691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:11:50.483177 31691 hybrid_clock.cc:648] HybridClock initialized: now 1786072310483152 us; error 50 us; skew 500 ppm
I20260807 03:11:50.483523 31691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:11:50 dist-test-slave-tm1g krb5kdc[31678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072310, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:50.486558 31691 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:11:50.487999 31691 webserver.cc:460] Webserver started at http://127.17.31.62:43217/ using document root <none> and password file <none>
I20260807 03:11:50.488677 31691 fs_manager.cc:359] Metadata directory not provided
I20260807 03:11:50.488821 31691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:11:50.489122 31691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:11:50.492020 31691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "1689540801ea46d1a85d802092b9d196"
format_stamp: "Formatted at 2026-08-07 03:11:50 on dist-test-slave-tm1g"
server_key: "b9a0187372fe59abe5d97ea74e751530"
server_key_iv: "7d565b0407aa93d2a19ade482ab0318b"
server_key_version: "encryptionkey@0"
I20260807 03:11:50.492810 31691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "1689540801ea46d1a85d802092b9d196"
format_stamp: "Formatted at 2026-08-07 03:11:50 on dist-test-slave-tm1g"
server_key: "b9a0187372fe59abe5d97ea74e751530"
server_key_iv: "7d565b0407aa93d2a19ade482ab0318b"
server_key_version: "encryptionkey@0"
I20260807 03:11:50.497737 31691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 03:11:50.501082 31708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:50.502300 31691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:11:50.502547 31691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "1689540801ea46d1a85d802092b9d196"
format_stamp: "Formatted at 2026-08-07 03:11:50 on dist-test-slave-tm1g"
server_key: "b9a0187372fe59abe5d97ea74e751530"
server_key_iv: "7d565b0407aa93d2a19ade482ab0318b"
server_key_version: "encryptionkey@0"
I20260807 03:11:50.502754 31691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:11:50.523891 31691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:11:50.527678 31691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:11:50.527959 31691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:11:50.546213 31691 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:39973
I20260807 03:11:50.546262 31764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:39973 every 8 connection(s)
I20260807 03:11:50.547581 31691 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:11:50.551119 31765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:11:50.557034 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 31691
I20260807 03:11:50.557277 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:11:50.557914 17532 external_mini_cluster.cc:1428] Setting key 938a325958d47381cff3548d645f3f1a
I20260807 03:11:50.557474 31765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:11:50 dist-test-slave-tm1g krb5kdc[31678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072310, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.31.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.61@KRBTEST.COM" created.
I20260807 03:11:50.586062 31767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:50.560654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45904 (local address 127.17.31.62:39973)
0807 03:11:50.561503 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:50.561520 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:50.561969 (+ 449us) server_negotiation.cc:407] Connection header received
0807 03:11:50.563960 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:50.563965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:50.564168 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:50.564373 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:50.565637 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:50.566963 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:50.568312 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:50.568857 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:50.569068 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:50.577697 (+ 8629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:50.577711 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:50.577730 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:50.577785 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:50.581583 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:50.582250 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:50.582256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:50.582258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:50.582535 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:50.582885 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:50.582888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:50.582890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:50.583355 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:50.583621 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:50.584204 (+ 583us) server_negotiation.cc:299] Negotiation successful
0807 03:11:50.584619 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":606,"thread_start_us":405,"threads_started":1}
I20260807 03:11:50.586083 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:50.560655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:39973 (local address 127.0.0.1:45904)
0807 03:11:50.561805 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:50.561836 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:50.563599 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:50.564386 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:50.564393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:50.565084 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:50.565438 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:50.565456 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:50.567129 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:50.567133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:50.568170 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:50.568182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:50.569090 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:50.569097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:50.569272 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:50.574685 (+ 5413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:50.574712 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:50.577429 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:50.581736 (+ 4307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:50.581746 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:50.581754 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:50.582130 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:50.582637 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:50.582640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:50.582643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:50.582793 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:50.583541 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:50.583554 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:50.583778 (+ 224us) client_negotiation.cc:769] Sending connection context
0807 03:11:50.584085 (+ 307us) client_negotiation.cc:240] Negotiation successful
0807 03:11:50.584728 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":210,"spinlock_wait_cycles":24832,"thread_start_us":744,"threads_started":1}
I20260807 03:11:50.589119 31765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:11:50.590227 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:50.589846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:45185 (local address 127.0.0.1:38090)
0807 03:11:50.590029 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:50.590126 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:45185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260807 03:11:50.590528 31709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.31.61:45185: connect: Connection refused (error 111)
I20260807 03:11:50.592100 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:50.591671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:45185 (local address 127.0.0.1:38106)
0807 03:11:50.591822 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:50.591917 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:45185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 03:11:50.592335 31765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.61:45185: Network error: Client connection negotiation failed: client connection to 127.17.31.61:45185: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.61.keytab.
Entry for principal kudu/127.17.31.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.61.keytab.
I20260807 03:11:50.612783 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.61:45185
--webserver_interface=127.17.31.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.61.keytab
--principal=kudu/127.17.31.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:39973,127.17.31.61:45185,127.17.31.60:39545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:11:50.641743 31765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } attempt: 1
I20260807 03:11:50.642947 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:50.642493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:45185 (local address 127.0.0.1:38116)
0807 03:11:50.642698 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:50.642814 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:45185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 03:11:50.644115 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:50.643744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:45185 (local address 127.0.0.1:38132)
0807 03:11:50.643934 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:50.644001 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:45185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 03:11:50.644373 31765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.61:45185: Network error: Client connection negotiation failed: client connection to 127.17.31.61:45185: connect: Connection refused (error 111)
I20260807 03:11:50.712606 31765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } attempt: 2
I20260807 03:11:50.713752 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:50.713329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:45185 (local address 127.0.0.1:38134)
0807 03:11:50.713521 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:50.713619 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:45185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260807 03:11:50.715068 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:50.714735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:45185 (local address 127.0.0.1:38138)
0807 03:11:50.714872 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:50.714939 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:45185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 03:11:50.715350 31765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.61:45185: Network error: Client connection negotiation failed: client connection to 127.17.31.61:45185: connect: Connection refused (error 111)
I20260807 03:11:50.806663 31765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } attempt: 3
I20260807 03:11:50.807912 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:50.807433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:45185 (local address 127.0.0.1:38140)
0807 03:11:50.807652 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:50.807754 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:45185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260807 03:11:50.808929 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:50.808553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:45185 (local address 127.0.0.1:38146)
0807 03:11:50.808698 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:50.808786 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:45185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 03:11:50.809115 31765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.61:45185: Network error: Client connection negotiation failed: client connection to 127.17.31.61:45185: connect: Connection refused (error 111)
W20260807 03:11:50.935461 31772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:11:50.935782 31772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:11:50.935953 31772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:11:50.942999 31772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:11:50.943136 31772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:11:50.943241 31772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:11:50.943359 31772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:11:50.943440 31772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:11:50.952358 31765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } attempt: 4
I20260807 03:11:50.953099 31772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:39973,127.17.31.61:45185,127.17.31.60:39545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.61.keytab
--principal=kudu/127.17.31.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.61:45185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.31.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:11:50.953576 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:50.953006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:45185 (local address 127.0.0.1:38158)
0807 03:11:50.953324 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:50.953445 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:45185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260807 03:11:50.954536 31772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:11:50.954571 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:50.954204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:45185 (local address 127.0.0.1:38164)
0807 03:11:50.954373 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:50.954458 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:45185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 03:11:50.954751 31765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.61:45185: Network error: Client connection negotiation failed: client connection to 127.17.31.61:45185: connect: Connection refused (error 111)
I20260807 03:11:50.955906 31772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:11:50.967262 31777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:50.967381 31781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:50.967381 31779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:50.968381 31772 server_base.cc:1034] running on GCE node
I20260807 03:11:50.969179 31772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:11:50.970614 31772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:11:50.971809 31772 hybrid_clock.cc:648] HybridClock initialized: now 1786072310971769 us; error 29 us; skew 500 ppm
I20260807 03:11:50.972144 31772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:11:50 dist-test-slave-tm1g krb5kdc[31678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072310, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:50.975232 31772 init.cc:377] Logged in from keytab as kudu/127.17.31.61@KRBTEST.COM (short username kudu)
I20260807 03:11:50.976759 31772 webserver.cc:460] Webserver started at http://127.17.31.61:40601/ using document root <none> and password file <none>
I20260807 03:11:50.977450 31772 fs_manager.cc:359] Metadata directory not provided
I20260807 03:11:50.977594 31772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:11:50.977901 31772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:11:50.980808 31772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/data/instance:
uuid: "cd7acafe83464a188dd7a7455420ed7b"
format_stamp: "Formatted at 2026-08-07 03:11:50 on dist-test-slave-tm1g"
server_key: "5553f4ec9f828d71df8014af6ad2f6ef"
server_key_iv: "cb480530a8bc4562c1da415d692b39a3"
server_key_version: "encryptionkey@0"
I20260807 03:11:50.981591 31772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/wal/instance:
uuid: "cd7acafe83464a188dd7a7455420ed7b"
format_stamp: "Formatted at 2026-08-07 03:11:50 on dist-test-slave-tm1g"
server_key: "5553f4ec9f828d71df8014af6ad2f6ef"
server_key_iv: "cb480530a8bc4562c1da415d692b39a3"
server_key_version: "encryptionkey@0"
I20260807 03:11:50.986529 31772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260807 03:11:50.989758 31788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:50.991066 31772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:11:50.991245 31772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/wal
uuid: "cd7acafe83464a188dd7a7455420ed7b"
format_stamp: "Formatted at 2026-08-07 03:11:50 on dist-test-slave-tm1g"
server_key: "5553f4ec9f828d71df8014af6ad2f6ef"
server_key_iv: "cb480530a8bc4562c1da415d692b39a3"
server_key_version: "encryptionkey@0"
I20260807 03:11:50.991441 31772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:11:51.007640 31772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:11:51.011264 31772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:11:51.011536 31772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:11:51.029207 31772 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.61:45185
I20260807 03:11:51.029251 31843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.61:45185 every 8 connection(s)
I20260807 03:11:51.030511 31772 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/data/info.pb
I20260807 03:11:51.033885 31844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:11:51.033946 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 31772
I20260807 03:11:51.034157 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-1/wal/instance
I20260807 03:11:51.034664 17532 external_mini_cluster.cc:1428] Setting key 7f79dec6b5a8a75bf5aa3e8540f8dcc5
I20260807 03:11:51.040258 31844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:11:51 dist-test-slave-tm1g krb5kdc[31678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072310, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.61@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.31.60@KRBTEST.COM; defaulting to no policy
I20260807 03:11:51.062637 31767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.042888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.17.31.62:39973)
0807 03:11:51.043070 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:51.043075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:51.044457 (+ 1382us) server_negotiation.cc:407] Connection header received
0807 03:11:51.046907 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:51.046912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:51.047015 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:51.047118 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:51.048564 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.049781 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.051412 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.052234 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.052405 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.056191 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:51.056197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:51.056200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:51.056231 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:51.059009 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.060026 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.060034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.060038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.060281 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.060862 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.060866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.060868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.061131 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:51.061239 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:51.062180 (+ 941us) server_negotiation.cc:299] Negotiation successful
0807 03:11:51.062416 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:11:51.063592 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.043476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:39973 (local address 127.0.0.1:45910)
0807 03:11:51.044301 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.044343 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:51.046657 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:51.047335 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:51.047343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:51.047811 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:51.048378 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:51.048394 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.049931 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.049935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:51.051266 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:51.051277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.052459 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.052464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:51.052643 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.053340 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:51.053361 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:51.055969 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:51.059149 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:51.059160 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:51.059178 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:51.059818 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:51.060427 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:51.060466 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:51.060472 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:51.060734 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:51.061274 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:51.061283 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:51.061751 (+ 468us) client_negotiation.cc:769] Sending connection context
0807 03:11:51.062052 (+ 301us) client_negotiation.cc:240] Negotiation successful
0807 03:11:51.062515 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":418,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.60@KRBTEST.COM" created.
I20260807 03:11:51.065270 31844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:11:51 dist-test-slave-tm1g krb5kdc[31678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072310, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.61@KRBTEST.COM for kudu/127.17.31.61@KRBTEST.COM
I20260807 03:11:51.082216 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.066051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:45185 (local address 127.0.0.1:38180)
0807 03:11:51.066228 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.066248 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:51.066369 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:51.067071 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:51.067076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:51.067568 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:51.067720 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:51.067730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.069141 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.069150 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:51.070986 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:51.071005 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.071877 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.071889 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:51.072170 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.072785 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:51.072811 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:51.075775 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:51.079960 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:51.079964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:51.079966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:51.080295 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:51.080696 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:51.080699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:51.080701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:51.080843 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:51.081579 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:51.081582 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:51.081706 (+ 124us) client_negotiation.cc:769] Sending connection context
0807 03:11:51.081805 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 03:11:51.081981 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 03:11:51.082844 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.066148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.17.31.61:45185)
0807 03:11:51.066715 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:51.066720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:51.066742 (+ 22us) server_negotiation.cc:407] Connection header received
0807 03:11:51.066779 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:51.066784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:51.066932 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:51.067082 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:51.067887 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.068977 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.071177 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.071709 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.071811 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.076033 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:51.076049 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:51.076066 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:51.076106 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:51.079841 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.080428 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.080437 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.080441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.080594 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.080962 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.080969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.080976 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.081442 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:51.081667 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:51.082023 (+ 356us) server_negotiation.cc:299] Negotiation successful
0807 03:11:51.082334 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":299,"threads_started":1}
I20260807 03:11:51.085042 31844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:11:51.086270 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.085721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44018)
0807 03:11:51.085941 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.086080 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260807 03:11:51.086580 31791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
I20260807 03:11:51.088286 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.087810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44032)
0807 03:11:51.088024 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.088132 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260807 03:11:51.088533 31844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:39545: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.60.keytab.
Entry for principal kudu/127.17.31.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.60.keytab.
I20260807 03:11:51.097350 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.60:39545
--webserver_interface=127.17.31.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.60.keytab
--principal=kudu/127.17.31.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:42013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:39973,127.17.31.61:45185,127.17.31.60:39545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:11:51.137888 31844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } attempt: 1
I20260807 03:11:51.139199 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.138631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44048)
0807 03:11:51.138856 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.138993 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 03:11:51.140267 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.139839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44056)
0807 03:11:51.139996 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.140099 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 03:11:51.140496 31844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:39545: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
I20260807 03:11:51.208696 31844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } attempt: 2
I20260807 03:11:51.209887 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.209338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44064)
0807 03:11:51.209579 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.209694 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 03:11:51.210994 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.210585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44070)
0807 03:11:51.210752 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.210837 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 03:11:51.211216 31844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:39545: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
I20260807 03:11:51.254061 31765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } attempt: 5
Aug 07 03:11:51 dist-test-slave-tm1g krb5kdc[31678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072310, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for kudu/127.17.31.61@KRBTEST.COM
I20260807 03:11:51.267803 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.255142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.17.31.61:45185)
0807 03:11:51.255344 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:51.255351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:51.255372 (+ 21us) server_negotiation.cc:407] Connection header received
0807 03:11:51.255533 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:51.255538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:51.255629 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:51.255740 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:51.256584 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.257281 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.258507 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.258996 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.259130 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.262481 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:51.262486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:51.262489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:51.262520 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:51.265192 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.266024 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.266028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.266030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.266143 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.266561 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.266564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.266566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.266836 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:51.266942 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:51.267426 (+ 484us) server_negotiation.cc:299] Negotiation successful
0807 03:11:51.267584 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 03:11:51.267930 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.254997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:45185 (local address 127.0.0.1:38192)
0807 03:11:51.255198 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.255222 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:51.255385 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:51.255784 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:51.255789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:51.256219 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:51.256407 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:51.256416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.257417 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.257424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:51.258293 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:51.258371 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.259157 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.259163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:51.259348 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.260026 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:51.260044 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:51.262276 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:51.265342 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:51.265347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:51.265351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:51.265895 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:51.266252 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:51.266256 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:51.266259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:51.266449 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:51.266977 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:51.266983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:51.267184 (+ 201us) client_negotiation.cc:769] Sending connection context
0807 03:11:51.267331 (+ 147us) client_negotiation.cc:240] Negotiation successful
0807 03:11:51.267601 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 03:11:51.269121 31765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:11:51.270433 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.269865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44078)
0807 03:11:51.270092 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.270200 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260807 03:11:51.271576 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.271150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44090)
0807 03:11:51.271325 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.271435 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 03:11:51.271828 31765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:39545: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
I20260807 03:11:51.302418 31844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } attempt: 3
I20260807 03:11:51.303689 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.303056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44096)
0807 03:11:51.303325 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.303467 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260807 03:11:51.304827 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.304363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44100)
0807 03:11:51.304593 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.304689 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260807 03:11:51.305047 31844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:39545: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
I20260807 03:11:51.323114 31765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } attempt: 1
I20260807 03:11:51.324327 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.323777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44108)
0807 03:11:51.324008 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.324117 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 03:11:51.325498 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.325060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44120)
0807 03:11:51.325237 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.325350 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 03:11:51.325726 31765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:39545: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
I20260807 03:11:51.394007 31765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } attempt: 2
I20260807 03:11:51.395370 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.394787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44128)
0807 03:11:51.395054 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.395175 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 03:11:51.396607 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.396193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44140)
0807 03:11:51.396355 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.396434 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 03:11:51.396874 31765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:39545: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
W20260807 03:11:51.441128 31851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:11:51.441406 31851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:11:51.441519 31851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260807 03:11:51.448308 31844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } attempt: 4
W20260807 03:11:51.448612 31851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:11:51.448772 31851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:11:51.448879 31851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:11:51.448967 31851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:11:51.449034 31851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:11:51.449609 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.449070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44144)
0807 03:11:51.449319 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.449427 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260807 03:11:51.450807 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.450427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44150)
0807 03:11:51.450594 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.450680 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 03:11:51.451022 31844 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:39545: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
I20260807 03:11:51.459626 31851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:42013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:39973,127.17.31.61:45185,127.17.31.60:39545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.60.keytab
--principal=kudu/127.17.31.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.60:39545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.31.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:11:51.460899 31851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:11:51.462157 31851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:11:51.473861 31856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:51.473897 31860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:51.473959 31857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:51.475147 31851 server_base.cc:1034] running on GCE node
I20260807 03:11:51.475929 31851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:11:51.477425 31851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:11:51.478672 31851 hybrid_clock.cc:648] HybridClock initialized: now 1786072311478624 us; error 54 us; skew 500 ppm
I20260807 03:11:51.479030 31851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:11:51 dist-test-slave-tm1g krb5kdc[31678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072311, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:51.481984 31851 init.cc:377] Logged in from keytab as kudu/127.17.31.60@KRBTEST.COM (short username kudu)
I20260807 03:11:51.483409 31851 webserver.cc:460] Webserver started at http://127.17.31.60:45689/ using document root <none> and password file <none>
I20260807 03:11:51.484074 31851 fs_manager.cc:359] Metadata directory not provided
I20260807 03:11:51.484220 31851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:11:51.484524 31851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:11:51.487491 31851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/data/instance:
uuid: "ea957fcdd980495ca3652c4532461187"
format_stamp: "Formatted at 2026-08-07 03:11:51 on dist-test-slave-tm1g"
server_key: "f5b86bef04a0c5fd8ca52699fb309deb"
server_key_iv: "7aa0eadf9c9705622d967bbac8fc3c5f"
server_key_version: "encryptionkey@0"
I20260807 03:11:51.488353 31851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/wal/instance:
uuid: "ea957fcdd980495ca3652c4532461187"
format_stamp: "Formatted at 2026-08-07 03:11:51 on dist-test-slave-tm1g"
server_key: "f5b86bef04a0c5fd8ca52699fb309deb"
server_key_iv: "7aa0eadf9c9705622d967bbac8fc3c5f"
server_key_version: "encryptionkey@0"
I20260807 03:11:51.493319 31851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:11:51.496788 31868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:51.498090 31851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:11:51.498296 31851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/wal
uuid: "ea957fcdd980495ca3652c4532461187"
format_stamp: "Formatted at 2026-08-07 03:11:51 on dist-test-slave-tm1g"
server_key: "f5b86bef04a0c5fd8ca52699fb309deb"
server_key_iv: "7aa0eadf9c9705622d967bbac8fc3c5f"
server_key_version: "encryptionkey@0"
I20260807 03:11:51.498551 31851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:11:51.503113 31765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } attempt: 3
I20260807 03:11:51.504303 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.503743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44156)
0807 03:11:51.504012 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.504114 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260807 03:11:51.505350 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.504932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44166)
0807 03:11:51.505127 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.505212 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260807 03:11:51.505563 31765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:39545: Network error: Client connection negotiation failed: client connection to 127.17.31.60:39545: connect: Connection refused (error 111)
I20260807 03:11:51.515102 31851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:11:51.518927 31851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:11:51.519160 31851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:11:51.537981 31851 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.60:39545
I20260807 03:11:51.538023 31920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.60:39545 every 8 connection(s)
I20260807 03:11:51.539283 31851 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/data/info.pb
I20260807 03:11:51.542742 31921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:11:51.549125 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 31851
I20260807 03:11:51.549347 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786072106928531-17532-0/minicluster-data/master-2/wal/instance
I20260807 03:11:51.549803 17532 external_mini_cluster.cc:1428] Setting key df9241c52e8aefd7a68f0cb3d11ab7c1
I20260807 03:11:51.550858 31921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:11:51 dist-test-slave-tm1g krb5kdc[31678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072311, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.60@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
Aug 07 03:11:51 dist-test-slave-tm1g krb5kdc[31678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.60@KRBTEST.COM
I20260807 03:11:51.570091 31767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.553597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45924 (local address 127.17.31.62:39973)
0807 03:11:51.553809 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:51.553818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:51.554741 (+ 923us) server_negotiation.cc:407] Connection header received
0807 03:11:51.555588 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:51.555593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:51.555674 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:51.555779 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:51.557158 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.557889 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.559525 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.559983 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.560113 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.563605 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:51.563613 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:51.563618 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:51.563663 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:51.566497 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.567549 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.567553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.567555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.567733 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.568252 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.568255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.568257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.568528 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:51.568636 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:51.569481 (+ 845us) server_negotiation.cc:299] Negotiation successful
0807 03:11:51.569689 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 03:11:51.570917 31925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.553652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:39973 (local address 127.0.0.1:45924)
0807 03:11:51.554596 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.554635 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:51.555346 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:51.555841 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:51.555851 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:51.556434 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:51.556991 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:51.557006 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.558062 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.558071 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:51.559387 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:51.559399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.560229 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.560236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:51.560438 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.561115 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:51.561142 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:51.563394 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:51.566654 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:51.566663 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:51.566670 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:51.567417 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:51.567877 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:51.567884 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:51.567889 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:51.568125 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:51.568657 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:51.568665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:51.569087 (+ 422us) client_negotiation.cc:769] Sending connection context
0807 03:11:51.569399 (+ 312us) client_negotiation.cc:240] Negotiation successful
0807 03:11:51.569827 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":482,"threads_started":1}
I20260807 03:11:51.572745 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.551644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44168 (local address 127.17.31.60:39545)
0807 03:11:51.552488 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:51.552502 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:51.552548 (+ 46us) server_negotiation.cc:407] Connection header received
0807 03:11:51.555336 (+ 2788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:51.555361 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:51.555633 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:51.557643 (+ 2010us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:51.557856 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.559157 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.560515 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.561020 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.561158 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.566000 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:51.566008 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:51.566018 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:51.566046 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:51.569546 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.570420 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.570426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.570428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.570561 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.570965 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.570971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.570974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.571452 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:51.571712 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:51.572152 (+ 440us) server_negotiation.cc:299] Negotiation successful
0807 03:11:51.572327 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":386,"threads_started":1}
I20260807 03:11:51.572665 31921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:11:51 dist-test-slave-tm1g krb5kdc[31678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072311, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.60@KRBTEST.COM for kudu/127.17.31.61@KRBTEST.COM
Aug 07 03:11:51 dist-test-slave-tm1g krb5kdc[31678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:11:51.587204 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.573479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38206 (local address 127.17.31.61:45185)
0807 03:11:51.573686 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:51.573693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:51.573893 (+ 200us) server_negotiation.cc:407] Connection header received
0807 03:11:51.574067 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:51.574073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:51.574146 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:51.574250 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:51.575118 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.576102 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.577499 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.577969 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.578099 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.581420 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:51.581428 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:51.581435 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:51.581481 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:51.584538 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.585323 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.585329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.585334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.585515 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.585926 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.585928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.585930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.586163 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:51.586348 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:51.586777 (+ 429us) server_negotiation.cc:299] Negotiation successful
0807 03:11:51.586943 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 03:11:51.587347 31925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.573487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:45185 (local address 127.0.0.1:38206)
0807 03:11:51.573804 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.573818 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:51.573951 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:51.574269 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:51.574273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:51.574813 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:51.574972 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:51.574980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.576234 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.576240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:51.577338 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:51.577352 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.578119 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.578123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:51.578285 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.578866 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:51.578881 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:51.581217 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:51.584660 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:51.584665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:51.584668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:51.585197 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:51.585632 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:51.585637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:51.585640 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:51.585814 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:51.586287 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:51.586291 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:51.586533 (+ 242us) client_negotiation.cc:769] Sending connection context
0807 03:11:51.586695 (+ 162us) client_negotiation.cc:240] Negotiation successful
0807 03:11:51.586989 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228}
I20260807 03:11:51.588435 31921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:11:51.590936 31767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.575321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.17.31.62:39973)
0807 03:11:51.575472 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:51.575477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:51.575494 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:11:51.575640 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:51.575645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:51.575709 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:51.575800 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:51.576898 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.577813 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.579194 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.579850 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.579998 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.584635 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:51.584641 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:51.584645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:51.584683 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:51.588066 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.588810 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.588815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.588818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.588984 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.589537 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.589543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.589548 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.589818 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:51.590120 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:51.590434 (+ 314us) server_negotiation.cc:299] Negotiation successful
0807 03:11:51.590653 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 07 03:11:51 dist-test-slave-tm1g krb5kdc[31678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072311, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.60@KRBTEST.COM for kudu/127.17.31.60@KRBTEST.COM
Aug 07 03:11:51 dist-test-slave-tm1g krb5kdc[31678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.61@KRBTEST.COM
I20260807 03:11:51.601660 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.589140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44180 (local address 127.17.31.60:39545)
0807 03:11:51.589300 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:51.589305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:51.589360 (+ 55us) server_negotiation.cc:407] Connection header received
0807 03:11:51.589523 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:51.589528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:51.589590 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:51.589683 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:51.590364 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.591214 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.592492 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.593143 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.593283 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.596341 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:51.596348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:51.596353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:51.596392 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:51.599140 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.599752 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.599759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.599762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.599881 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.600221 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.600228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.600231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.600501 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:51.600646 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:51.600988 (+ 342us) server_negotiation.cc:299] Negotiation successful
0807 03:11:51.601337 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":53}
I20260807 03:11:51.601660 31925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.589131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44180)
0807 03:11:51.589299 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.589318 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:51.589411 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:51.589697 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:51.589701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:51.590074 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:51.590209 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:51.590217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.591327 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.591334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:51.592346 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:51.592358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.593275 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.593279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:51.593424 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.593919 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:51.593939 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:51.596163 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:51.599263 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:51.599269 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:51.599273 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:51.599639 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:51.599983 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:51.599990 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:51.599996 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:51.600122 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:51.600614 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:51.600618 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:51.600755 (+ 137us) client_negotiation.cc:769] Sending connection context
0807 03:11:51.600903 (+ 148us) client_negotiation.cc:240] Negotiation successful
0807 03:11:51.601182 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 03:11:51.607650 31921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187: Bootstrap starting.
I20260807 03:11:51.608673 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.592540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.17.31.61:45185)
0807 03:11:51.592705 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:51.592709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:51.592743 (+ 34us) server_negotiation.cc:407] Connection header received
0807 03:11:51.592906 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:51.592909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:51.592966 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:51.593037 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:51.594605 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.595773 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.597416 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.597831 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.597924 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.602199 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:51.602205 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:51.602210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:51.602242 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:51.604836 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.605655 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.605658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.605660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.605826 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.606337 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.606342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.606345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.606697 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:51.607009 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:51.608103 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0807 03:11:51.608341 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 03:11:51.611835 31921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187: Neither blocks nor log segments found. Creating new log.
I20260807 03:11:51.613099 31921 log.cc:824] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187: Log is configured to *not* fsync() on all Append() calls
I20260807 03:11:51.613591 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:11:51.615793 31921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187: No bootstrap required, opened a new log
I20260807 03:11:51.628862 31921 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } }
I20260807 03:11:51.629446 31921 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:11:51.629603 31921 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea957fcdd980495ca3652c4532461187, State: Initialized, Role: FOLLOWER
I20260807 03:11:51.630503 31921 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } }
I20260807 03:11:51.632377 31935 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } } }
I20260807 03:11:51.632869 31935 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:11:51.633764 31921 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:11:51.646117 31848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.618848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.17.31.61:45185)
0807 03:11:51.619040 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:51.619047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:51.626885 (+ 7838us) server_negotiation.cc:407] Connection header received
0807 03:11:51.627100 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:51.627109 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:51.627206 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:51.627317 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:51.630140 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.631230 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.633420 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.634066 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.634163 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.638557 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:51.638565 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:51.638572 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:51.638614 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:51.642618 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.643506 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.643512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.643516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.643665 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.644152 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.644159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.644165 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.644571 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:51.644789 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:51.645482 (+ 693us) server_negotiation.cc:299] Negotiation successful
0807 03:11:51.645729 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 03:11:51.649364 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.618508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.17.31.60:39545)
0807 03:11:51.618722 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:51.618730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:51.625927 (+ 7197us) server_negotiation.cc:407] Connection header received
0807 03:11:51.626451 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:51.626460 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:51.626549 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:51.626627 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:51.629990 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.631210 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.634907 (+ 3697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.635725 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.635830 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.640608 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:51.640617 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:51.640625 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:51.640664 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:51.644495 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.646586 (+ 2091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.646592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.646595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.646777 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.647242 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.647253 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.647258 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.647668 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:51.647934 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:51.648668 (+ 734us) server_negotiation.cc:299] Negotiation successful
0807 03:11:51.648917 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 03:11:51.653641 31767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.618408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.17.31.62:39973)
0807 03:11:51.618840 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:51.618848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:51.629317 (+ 10469us) server_negotiation.cc:407] Connection header received
0807 03:11:51.629630 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:51.629637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:51.629726 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:51.629807 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:51.632442 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.633501 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.636127 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.637029 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.637151 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.646266 (+ 9115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:51.646275 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:51.646281 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:51.646359 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:51.650480 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.651387 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.651392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.651397 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.651586 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.652058 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.652063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.652066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.652362 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:51.652530 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:51.653000 (+ 470us) server_negotiation.cc:299] Negotiation successful
0807 03:11:51.653233 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260807 03:11:51.657143 31947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:11:51.657261 31947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:11:51.648659 31851 master_runner.cc:186] Error getting master registration for 127.17.31.62:39973: OK, instance_id { permanent_uuid: "1689540801ea46d1a85d802092b9d196" instance_seqno: 1786072310540983 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 03:11:51.660482 31851 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 03:11:51.682754 31765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } attempt: 4
Aug 07 03:11:51 dist-test-slave-tm1g krb5kdc[31678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072310, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for kudu/127.17.31.60@KRBTEST.COM
I20260807 03:11:51.698468 31768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.683312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44204)
0807 03:11:51.683506 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.683528 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:51.683635 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:51.683973 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:51.683978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:51.684373 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:51.684530 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:51.684538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.685722 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.685730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:51.687078 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:51.687094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.688144 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.688150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:51.688345 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.688980 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:51.689004 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:51.691546 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:51.694831 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:51.694837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:51.694841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:51.696435 (+ 1594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:51.696873 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:51.696877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:51.696879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:51.697016 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:51.697640 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:51.697643 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:51.697781 (+ 138us) client_negotiation.cc:769] Sending connection context
0807 03:11:51.697889 (+ 108us) client_negotiation.cc:240] Negotiation successful
0807 03:11:51.698109 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260807 03:11:51.698640 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.683425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44204 (local address 127.17.31.60:39545)
0807 03:11:51.683592 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:51.683598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:51.683616 (+ 18us) server_negotiation.cc:407] Connection header received
0807 03:11:51.683766 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:51.683771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:51.683846 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:51.683947 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:51.684674 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.685585 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.687249 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.687990 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.688089 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.691737 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:51.691742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:51.691745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:51.691774 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:51.694697 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.696553 (+ 1856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.696558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.696562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.696739 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.697131 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.697136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.697140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.697526 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:51.697677 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:51.698009 (+ 332us) server_negotiation.cc:299] Negotiation successful
0807 03:11:51.698238 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 03:11:51.703280 31765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196: Bootstrap starting.
I20260807 03:11:51.706933 31765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196: Neither blocks nor log segments found. Creating new log.
I20260807 03:11:51.708105 31765 log.cc:824] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196: Log is configured to *not* fsync() on all Append() calls
I20260807 03:11:51.710737 31765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196: No bootstrap required, opened a new log
I20260807 03:11:51.718596 31765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } }
I20260807 03:11:51.718868 31765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:11:51.719030 31765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1689540801ea46d1a85d802092b9d196, State: Initialized, Role: FOLLOWER
I20260807 03:11:51.719615 31765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } }
I20260807 03:11:51.720563 31950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } } }
I20260807 03:11:51.720952 31950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:11:51.721683 31765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 03:11:51.731384 31961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:11:51.731549 31961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:11:51.731623 31691 master_runner.cc:186] Error getting master registration for 127.17.31.61:45185: OK, instance_id { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" instance_seqno: 1786072311024083 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 03:11:51.739507 31691 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 03:11:51.750237 31844 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } attempt: 5
Aug 07 03:11:51 dist-test-slave-tm1g krb5kdc[31678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072310, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.61@KRBTEST.COM for kudu/127.17.31.60@KRBTEST.COM
I20260807 03:11:51.765362 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.750860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:39545 (local address 127.0.0.1:44214)
0807 03:11:51.751097 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:51.751116 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:51.751231 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:51.751574 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:51.751579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:51.752016 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:51.752172 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:51.752181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.753236 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.753242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:51.754129 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:51.754140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.754916 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.754922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:51.755122 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.755725 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:51.755742 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:51.757750 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:51.762363 (+ 4613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:51.762368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:51.762371 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:51.762915 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:51.763691 (+ 776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:51.763695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:51.763697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:51.763827 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:51.764342 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:51.764347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:51.764502 (+ 155us) client_negotiation.cc:769] Sending connection context
0807 03:11:51.764645 (+ 143us) client_negotiation.cc:240] Negotiation successful
0807 03:11:51.764937 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260807 03:11:51.765383 31924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:51.750911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44214 (local address 127.17.31.60:39545)
0807 03:11:51.751100 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:51.751106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:51.751168 (+ 62us) server_negotiation.cc:407] Connection header received
0807 03:11:51.751364 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:51.751369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:51.751445 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:51.751552 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:51.752317 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.753082 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.754273 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:51.754775 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:51.754909 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:51.757943 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:51.757949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:51.757953 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:51.757987 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:51.762179 (+ 4192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.763384 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.763389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.763393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.763551 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:51.763935 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:51.763941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:51.763944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:51.764221 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:51.764379 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:51.764756 (+ 377us) server_negotiation.cc:299] Negotiation successful
0807 03:11:51.764992 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 03:11:51.770224 31844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b: Bootstrap starting.
W20260807 03:11:51.771327 31875 tablet.cc:2369] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:11:51.773772 31844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b: Neither blocks nor log segments found. Creating new log.
I20260807 03:11:51.774868 31844 log.cc:824] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b: Log is configured to *not* fsync() on all Append() calls
I20260807 03:11:51.777623 31844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b: No bootstrap required, opened a new log
W20260807 03:11:51.780640 31715 tablet.cc:2369] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:11:51.786861 31844 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } }
I20260807 03:11:51.787197 31844 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:11:51.787343 31844 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7acafe83464a188dd7a7455420ed7b, State: Initialized, Role: FOLLOWER
I20260807 03:11:51.787963 31844 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } }
I20260807 03:11:51.788938 31964 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } } }
I20260807 03:11:51.789384 31964 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:11:51.790107 31844 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b [sys.catalog]: configured and running, proceeding with master startup.
W20260807 03:11:51.803540 31976 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:11:51.803757 31976 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:11:51.804656 31772 master_runner.cc:263] No leader master found from master cd7acafe83464a188dd7a7455420ed7b
I20260807 03:11:51.804792 31772 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 03:11:51.879102 31950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:11:51.879432 31950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } }
I20260807 03:11:51.881116 31950 leader_election.cc:288] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd7acafe83464a188dd7a7455420ed7b (127.17.31.61:45185), ea957fcdd980495ca3652c4532461187 (127.17.31.60:39545)
I20260807 03:11:51.882158 31896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1689540801ea46d1a85d802092b9d196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea957fcdd980495ca3652c4532461187" is_pre_election: true
I20260807 03:11:51.882772 31896 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1689540801ea46d1a85d802092b9d196 in term 0.
I20260807 03:11:51.883430 31711 leader_election.cc:302] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1689540801ea46d1a85d802092b9d196, ea957fcdd980495ca3652c4532461187; no voters:
I20260807 03:11:51.884539 31816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1689540801ea46d1a85d802092b9d196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7acafe83464a188dd7a7455420ed7b" is_pre_election: true
I20260807 03:11:51.885128 31816 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1689540801ea46d1a85d802092b9d196 in term 0.
I20260807 03:11:51.888015 31950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:11:51.888176 31950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:11:51.888271 31950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:11:51.893605 31950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } }
I20260807 03:11:51.894665 31816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1689540801ea46d1a85d802092b9d196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7acafe83464a188dd7a7455420ed7b"
I20260807 03:11:51.894891 31816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:11:51.898787 31816 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1689540801ea46d1a85d802092b9d196 in term 1.
I20260807 03:11:51.899200 31709 leader_election.cc:302] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1689540801ea46d1a85d802092b9d196, cd7acafe83464a188dd7a7455420ed7b; no voters:
I20260807 03:11:51.901062 31896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1689540801ea46d1a85d802092b9d196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea957fcdd980495ca3652c4532461187"
I20260807 03:11:51.901278 31896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:11:51.904773 31896 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1689540801ea46d1a85d802092b9d196 in term 1.
I20260807 03:11:51.905457 31950 leader_election.cc:288] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [CANDIDATE]: Term 1 election: Requested vote from peers cd7acafe83464a188dd7a7455420ed7b (127.17.31.61:45185), ea957fcdd980495ca3652c4532461187 (127.17.31.60:39545)
I20260807 03:11:51.905994 31950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:11:51.906224 31950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [term 1 LEADER]: Becoming Leader. State: Replica: 1689540801ea46d1a85d802092b9d196, State: Running, Role: LEADER
I20260807 03:11:51.907689 31950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } }
I20260807 03:11:51.925474 31978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1689540801ea46d1a85d802092b9d196. Latest consensus state: current_term: 1 leader_uuid: "1689540801ea46d1a85d802092b9d196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } } }
I20260807 03:11:51.925905 31978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [sys.catalog]: This master's current role is: LEADER
I20260807 03:11:51.928874 31980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:11:51.931152 31980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:11:51.947311 31896 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187 [term 1 FOLLOWER]: Refusing update from remote peer 1689540801ea46d1a85d802092b9d196: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:11:51.948370 31978 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:11:51.959452 31816 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b [term 1 FOLLOWER]: Refusing update from remote peer 1689540801ea46d1a85d802092b9d196: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:11:51.960325 31978 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:11:51.983682 31935 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1689540801ea46d1a85d802092b9d196. Latest consensus state: current_term: 1 leader_uuid: "1689540801ea46d1a85d802092b9d196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } } }
I20260807 03:11:51.984074 31935 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:11:51.989403 31982 mvcc.cc:205] Tried to move back new op lower bound from 7315752189723258880 to 7315752189584125952. Current Snapshot: MvccSnapshot[applied={T|T < 7315752189723258880}]
I20260807 03:11:51.991011 31981 mvcc.cc:205] Tried to move back new op lower bound from 7315752189723258880 to 7315752189584125952. Current Snapshot: MvccSnapshot[applied={T|T < 7315752189723258880}]
I20260807 03:11:51.999065 31950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1689540801ea46d1a85d802092b9d196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } } }
I20260807 03:11:51.999377 31950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [sys.catalog]: This master's current role is: LEADER
I20260807 03:11:52.001286 31980 catalog_manager.cc:1347] Generated new cluster ID: f5b9a885291d47eab50f604a7836a79c
I20260807 03:11:52.001387 31980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:11:52.009739 31935 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1689540801ea46d1a85d802092b9d196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } } }
I20260807 03:11:52.009981 31935 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187 [sys.catalog]: This master's current role is: FOLLOWER
W20260807 03:11:52.013902 31795 tablet.cc:2369] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:11:52.020174 31980 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:11:52.031021 31980 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:11:52.039093 31964 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1689540801ea46d1a85d802092b9d196. Latest consensus state: current_term: 1 leader_uuid: "1689540801ea46d1a85d802092b9d196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } } }
I20260807 03:11:52.039441 31964 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:11:52.046056 31980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196: Generated new TSK 0
I20260807 03:11:52.047060 31980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:11:52.049077 31950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1689540801ea46d1a85d802092b9d196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } } }
I20260807 03:11:52.049377 31950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1689540801ea46d1a85d802092b9d196 [sys.catalog]: This master's current role is: LEADER
I20260807 03:11:52.091639 31964 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1689540801ea46d1a85d802092b9d196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1689540801ea46d1a85d802092b9d196" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 39973 } } peers { permanent_uuid: "cd7acafe83464a188dd7a7455420ed7b" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 45185 } } peers { permanent_uuid: "ea957fcdd980495ca3652c4532461187" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 39545 } } }
I20260807 03:11:52.091902 31964 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:11:52.659279 31947 catalog_manager.cc:1259] Loaded cluster ID: f5b9a885291d47eab50f604a7836a79c
I20260807 03:11:52.659425 31947 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187: loading cluster ID for follower catalog manager: success
I20260807 03:11:52.661834 31947 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187: acquiring CA information for follower catalog manager: success
I20260807 03:11:52.663618 31947 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ea957fcdd980495ca3652c4532461187: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 03:11:52.805927 31976 catalog_manager.cc:1259] Loaded cluster ID: f5b9a885291d47eab50f604a7836a79c
I20260807 03:11:52.806037 31976 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b: loading cluster ID for follower catalog manager: success
I20260807 03:11:52.808705 31976 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b: acquiring CA information for follower catalog manager: success
I20260807 03:11:52.810464 31976 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd7acafe83464a188dd7a7455420ed7b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 03:11:53.188779 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 31691
I20260807 03:11:53.190162 31759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:11:53.301322 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 31691
I20260807 03:11:53.323724 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 31772
I20260807 03:11:53.324893 31838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:11:53.433493 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 31772
I20260807 03:11:53.455718 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 31851
I20260807 03:11:53.456873 31916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:11:53.567409 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 31851
2026-08-07T03:11:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4796 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:11:53 dist-test-slave-tm1g krb5kdc[32000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:11:53 dist-test-slave-tm1g krb5kdc[32000](info): set up 2 sockets
Aug 07 03:11:53 dist-test-slave-tm1g krb5kdc[32000](info): commencing operation
krb5kdc: starting...
W20260807 03:11:54.696715 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:11:54 dist-test-slave-tm1g krb5kdc[32000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:11:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:11:54.933590 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:34965
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:34965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:11:55.256085 32015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:11:55.256405 32015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:11:55.256573 32015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:11:55.263453 32015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:11:55.263597 32015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:11:55.263702 32015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:11:55.263805 32015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:11:55.263897 32015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:11:55.273185 32015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:34965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:34965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:11:55.274453 32015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:11:55.275749 32015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:11:55.287298 32023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:55.287370 32024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:55.287262 32026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:55.288369 32015 server_base.cc:1034] running on GCE node
I20260807 03:11:55.289104 32015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:11:55.290446 32015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:11:55.291682 32015 hybrid_clock.cc:648] HybridClock initialized: now 1786072315291629 us; error 53 us; skew 500 ppm
I20260807 03:11:55.292021 32015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:11:55 dist-test-slave-tm1g krb5kdc[32000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072315, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:55.295055 32015 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:11:55.296523 32015 webserver.cc:460] Webserver started at http://127.17.31.62:38441/ using document root <none> and password file <none>
I20260807 03:11:55.297187 32015 fs_manager.cc:359] Metadata directory not provided
I20260807 03:11:55.297335 32015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:11:55.297640 32015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:11:55.300504 32015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "a1e95f24c7334511a4efc9990b00baea"
format_stamp: "Formatted at 2026-08-07 03:11:55 on dist-test-slave-tm1g"
server_key: "8147eea69f0dffa26054871f7f36fe78"
server_key_iv: "df46878497d424a43004b2906b89d0cb"
server_key_version: "encryptionkey@0"
I20260807 03:11:55.301234 32015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "a1e95f24c7334511a4efc9990b00baea"
format_stamp: "Formatted at 2026-08-07 03:11:55 on dist-test-slave-tm1g"
server_key: "8147eea69f0dffa26054871f7f36fe78"
server_key_iv: "df46878497d424a43004b2906b89d0cb"
server_key_version: "encryptionkey@0"
I20260807 03:11:55.306113 32015 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260807 03:11:55.309377 32033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:55.310663 32015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:11:55.310846 32015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "a1e95f24c7334511a4efc9990b00baea"
format_stamp: "Formatted at 2026-08-07 03:11:55 on dist-test-slave-tm1g"
server_key: "8147eea69f0dffa26054871f7f36fe78"
server_key_iv: "df46878497d424a43004b2906b89d0cb"
server_key_version: "encryptionkey@0"
I20260807 03:11:55.311036 32015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:11:55.324679 32015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:11:55.328725 32015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:11:55.329008 32015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:11:55.346946 32015 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:34965
I20260807 03:11:55.346964 32084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:34965 every 8 connection(s)
I20260807 03:11:55.348240 32015 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:11:55.351699 32085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:11:55.355080 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 32015
I20260807 03:11:55.355306 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:11:55.356158 17532 external_mini_cluster.cc:1428] Setting key ab6dc48cb527d5884a7ead35551cd452
I20260807 03:11:55.363541 32085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea: Bootstrap starting.
I20260807 03:11:55.366405 32085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea: Neither blocks nor log segments found. Creating new log.
I20260807 03:11:55.367332 32085 log.cc:824] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea: Log is configured to *not* fsync() on all Append() calls
Aug 07 03:11:55 dist-test-slave-tm1g krb5kdc[32000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:11:55.370245 32085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea: No bootstrap required, opened a new log
I20260807 03:11:55.378402 32088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:55.358018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.17.31.62:34965)
0807 03:11:55.358746 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:55.358756 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:55.358793 (+ 37us) server_negotiation.cc:407] Connection header received
0807 03:11:55.360385 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:55.360406 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:55.360763 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:55.361148 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:55.362635 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:55.363968 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:55.365267 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:55.365913 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:55.366108 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:55.371167 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:55.371174 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:55.371188 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:55.371226 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:55.374408 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:55.375203 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:55.375209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:55.375214 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:55.375386 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:55.375797 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:55.375803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:55.375805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:55.376296 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:55.376523 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:55.376980 (+ 457us) server_negotiation.cc:299] Negotiation successful
0807 03:11:55.377417 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":344,"threads_started":1}
I20260807 03:11:55.379029 32085 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e95f24c7334511a4efc9990b00baea" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34965 } }
I20260807 03:11:55.379395 32085 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:11:55.379510 32085 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e95f24c7334511a4efc9990b00baea, State: Initialized, Role: FOLLOWER
I20260807 03:11:55.380069 32085 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e95f24c7334511a4efc9990b00baea" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34965 } }
I20260807 03:11:55.380324 32085 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:11:55.380477 32085 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:11:55.380668 32085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:11:55.382228 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:11:55.383464 32085 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e95f24c7334511a4efc9990b00baea" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34965 } }
I20260807 03:11:55.383986 32085 leader_election.cc:302] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1e95f24c7334511a4efc9990b00baea; no voters:
I20260807 03:11:55.384562 32085 leader_election.cc:288] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:11:55.384692 32090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:11:55.386723 32090 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [term 1 LEADER]: Becoming Leader. State: Replica: a1e95f24c7334511a4efc9990b00baea, State: Running, Role: LEADER
I20260807 03:11:55.387189 32085 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:11:55.387207 32090 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e95f24c7334511a4efc9990b00baea" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34965 } }
I20260807 03:11:55.389897 32093 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1e95f24c7334511a4efc9990b00baea. Latest consensus state: current_term: 1 leader_uuid: "a1e95f24c7334511a4efc9990b00baea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e95f24c7334511a4efc9990b00baea" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34965 } } }
I20260807 03:11:55.390568 32093 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [sys.catalog]: This master's current role is: LEADER
I20260807 03:11:55.391535 32091 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1e95f24c7334511a4efc9990b00baea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e95f24c7334511a4efc9990b00baea" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34965 } } }
I20260807 03:11:55.391863 32098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:11:55.391862 32091 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea [sys.catalog]: This master's current role is: LEADER
I20260807 03:11:55.396745 32098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:11:55.405586 32098 catalog_manager.cc:1347] Generated new cluster ID: d663fd19e70042a583950a9e67fc9c97
I20260807 03:11:55.405709 32098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:11:55.429505 32098 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:11:55.430820 32098 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:11:55.437430 32098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1e95f24c7334511a4efc9990b00baea: Generated new TSK 0
I20260807 03:11:55.438445 32098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:11:55.495056 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 32015
I20260807 03:11:55.496371 32080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:11:55.591702 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 32015
2026-08-07T03:11:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2019 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:11:55 dist-test-slave-tm1g krb5kdc[32114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:11:55 dist-test-slave-tm1g krb5kdc[32114](info): set up 2 sockets
Aug 07 03:11:55 dist-test-slave-tm1g krb5kdc[32114](info): commencing operation
krb5kdc: starting...
W20260807 03:11:56.708549 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:11:56 dist-test-slave-tm1g krb5kdc[32114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:11:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:11:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:11:56.965237 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:35575
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:35575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:11:57.282657 32127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:11:57.282905 32127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:11:57.283058 32127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:11:57.290108 32127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:11:57.290197 32127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:11:57.290328 32127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:11:57.290426 32127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:11:57.290501 32127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:11:57.299795 32127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:35575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:35575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:11:57.301021 32127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:11:57.302228 32127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:11:57.313316 32132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:57.313387 32133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:57.313429 32135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:57.314492 32127 server_base.cc:1034] running on GCE node
I20260807 03:11:57.315167 32127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:11:57.316433 32127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:11:57.317667 32127 hybrid_clock.cc:648] HybridClock initialized: now 1786072317317631 us; error 54 us; skew 500 ppm
I20260807 03:11:57.317993 32127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:11:57 dist-test-slave-tm1g krb5kdc[32114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072317, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:57.321203 32127 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:11:57.322775 32127 webserver.cc:460] Webserver started at http://127.17.31.62:44067/ using document root <none> and password file <none>
I20260807 03:11:57.323434 32127 fs_manager.cc:359] Metadata directory not provided
I20260807 03:11:57.323585 32127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:11:57.323889 32127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:11:57.326766 32127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "d2f1edcbb028476eaf98f25d4ea22649"
format_stamp: "Formatted at 2026-08-07 03:11:57 on dist-test-slave-tm1g"
server_key: "e5ea7860541af4f8c260d7a32513e249"
server_key_iv: "45d7c91a37cabc3da23859209ff9886c"
server_key_version: "encryptionkey@0"
I20260807 03:11:57.327513 32127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "d2f1edcbb028476eaf98f25d4ea22649"
format_stamp: "Formatted at 2026-08-07 03:11:57 on dist-test-slave-tm1g"
server_key: "e5ea7860541af4f8c260d7a32513e249"
server_key_iv: "45d7c91a37cabc3da23859209ff9886c"
server_key_version: "encryptionkey@0"
I20260807 03:11:57.332379 32127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260807 03:11:57.335680 32142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:57.336861 32127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 03:11:57.337039 32127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "d2f1edcbb028476eaf98f25d4ea22649"
format_stamp: "Formatted at 2026-08-07 03:11:57 on dist-test-slave-tm1g"
server_key: "e5ea7860541af4f8c260d7a32513e249"
server_key_iv: "45d7c91a37cabc3da23859209ff9886c"
server_key_version: "encryptionkey@0"
I20260807 03:11:57.337225 32127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:11:57.382910 32127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:11:57.386788 32127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:11:57.387094 32127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:11:57.405426 32127 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:35575
I20260807 03:11:57.405463 32196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:35575 every 8 connection(s)
I20260807 03:11:57.406780 32127 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:11:57.410368 32197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:11:57.415285 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 32127
I20260807 03:11:57.415513 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:11:57.416153 17532 external_mini_cluster.cc:1428] Setting key cfc0524a7e30ded2e84afd890f39c863
I20260807 03:11:57.421787 32197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649: Bootstrap starting.
I20260807 03:11:57.426147 32197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649: Neither blocks nor log segments found. Creating new log.
I20260807 03:11:57.427477 32197 log.cc:824] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649: Log is configured to *not* fsync() on all Append() calls
I20260807 03:11:57.431047 32197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649: No bootstrap required, opened a new log
Aug 07 03:11:57 dist-test-slave-tm1g krb5kdc[32114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:11:57.441109 32197 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } }
I20260807 03:11:57.441552 32197 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:11:57.441660 32197 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f1edcbb028476eaf98f25d4ea22649, State: Initialized, Role: FOLLOWER
I20260807 03:11:57.442320 32197 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } }
I20260807 03:11:57.442611 32197 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:11:57.442799 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:57.418172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.17.31.62:35575)
0807 03:11:57.419009 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:57.419024 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:57.419063 (+ 39us) server_negotiation.cc:407] Connection header received
0807 03:11:57.421060 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:57.421088 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:57.421630 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:57.422084 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:57.424167 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:57.426339 (+ 2172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:57.428508 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:57.429593 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:57.429772 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:57.434282 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:57.434293 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:57.434330 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:57.434384 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:57.438368 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:57.438995 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:57.439005 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:57.439015 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:57.439235 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:57.439580 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:57.439585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:57.439590 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:57.440300 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:57.440601 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:57.440963 (+ 362us) server_negotiation.cc:299] Negotiation successful
0807 03:11:57.441388 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":379,"threads_started":1}
I20260807 03:11:57.442776 32197 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:11:57.442974 32197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:11:57.445248 32197 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } }
I20260807 03:11:57.445794 32197 leader_election.cc:302] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2f1edcbb028476eaf98f25d4ea22649; no voters:
I20260807 03:11:57.446394 32197 leader_election.cc:288] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:11:57.446548 32202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:11:57.447280 32202 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 1 LEADER]: Becoming Leader. State: Replica: d2f1edcbb028476eaf98f25d4ea22649, State: Running, Role: LEADER
I20260807 03:11:57.447923 32202 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } }
I20260807 03:11:57.448457 32197 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:11:57.451731 32204 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2f1edcbb028476eaf98f25d4ea22649. Latest consensus state: current_term: 1 leader_uuid: "d2f1edcbb028476eaf98f25d4ea22649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } } }
I20260807 03:11:57.451704 32203 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2f1edcbb028476eaf98f25d4ea22649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } } }
I20260807 03:11:57.452221 32204 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [sys.catalog]: This master's current role is: LEADER
I20260807 03:11:57.452247 32203 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [sys.catalog]: This master's current role is: LEADER
I20260807 03:11:57.455586 32212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:11:57.460379 32212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:11:57.469389 32212 catalog_manager.cc:1347] Generated new cluster ID: 58655f54010242ab891eb566c481737a
I20260807 03:11:57.469515 32212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:11:57.490204 32212 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:11:57.492085 32212 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:11:57.501148 32212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649: Generated new TSK 0
I20260807 03:11:57.502141 32212 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:11:57.578608 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35575
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:11:57.922175 32223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:11:57.922519 32223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:11:57.922683 32223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:11:57.929795 32223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:11:57.929940 32223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:11:57.930109 32223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:11:57.939544 32223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:11:57.940806 32223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:11:57.942162 32223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:11:57.956117 32232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:57.956211 32228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:57.956184 32229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:57.956871 32223 server_base.cc:1034] running on GCE node
I20260807 03:11:57.957491 32223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:11:57.958416 32223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:11:57.959627 32223 hybrid_clock.cc:648] HybridClock initialized: now 1786072317959582 us; error 39 us; skew 500 ppm
I20260807 03:11:57.959936 32223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:11:57 dist-test-slave-tm1g krb5kdc[32114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072317, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:57.963033 32223 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:11:57.964366 32223 webserver.cc:460] Webserver started at http://127.17.31.1:34937/ using document root <none> and password file <none>
I20260807 03:11:57.965045 32223 fs_manager.cc:359] Metadata directory not provided
I20260807 03:11:57.965202 32223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:11:57.965505 32223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:11:57.968327 32223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "fbc8f9b4702f43ae934d6eae53ac2bae"
format_stamp: "Formatted at 2026-08-07 03:11:57 on dist-test-slave-tm1g"
server_key: "a9b07e2c577b875865616afd95643ebb"
server_key_iv: "5ca80a34344e7985807784a496a241ef"
server_key_version: "encryptionkey@0"
I20260807 03:11:57.969103 32223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "fbc8f9b4702f43ae934d6eae53ac2bae"
format_stamp: "Formatted at 2026-08-07 03:11:57 on dist-test-slave-tm1g"
server_key: "a9b07e2c577b875865616afd95643ebb"
server_key_iv: "5ca80a34344e7985807784a496a241ef"
server_key_version: "encryptionkey@0"
I20260807 03:11:57.973996 32223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 03:11:57.977201 32239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:57.978504 32223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 03:11:57.978681 32223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "fbc8f9b4702f43ae934d6eae53ac2bae"
format_stamp: "Formatted at 2026-08-07 03:11:57 on dist-test-slave-tm1g"
server_key: "a9b07e2c577b875865616afd95643ebb"
server_key_iv: "5ca80a34344e7985807784a496a241ef"
server_key_version: "encryptionkey@0"
I20260807 03:11:57.978883 32223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:11:58.008370 32223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:11:58.012594 32223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:11:58.012887 32223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:11:58.013788 32223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:11:58.015177 32223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:11:58.015280 32223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:58.015430 32223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:11:58.015515 32223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:58.048507 32223 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:45557
I20260807 03:11:58.048595 32357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:45557 every 8 connection(s)
I20260807 03:11:58.049808 32223 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:11:58.056481 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 32223
I20260807 03:11:58.056684 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:11:58.057346 17532 external_mini_cluster.cc:1428] Setting key 839a54067d51ad724f4b40d7bf4e1491
Aug 07 03:11:58 dist-test-slave-tm1g krb5kdc[32114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072317, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:11:58.071587 32360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:58.052353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:51993 (local address 127.17.31.62:35575)
0807 03:11:58.052834 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:58.052838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:58.053591 (+ 753us) server_negotiation.cc:407] Connection header received
0807 03:11:58.055448 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:58.055454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:58.055542 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:58.055664 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:58.057163 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:58.058029 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:58.060050 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:58.060435 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:58.060535 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:58.064573 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:58.064581 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:58.064589 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:58.064634 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:58.068461 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:58.069219 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:58.069226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:58.069232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:58.069431 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:58.069868 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:58.069875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:58.069881 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:58.070288 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:58.070499 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:58.071143 (+ 644us) server_negotiation.cc:299] Negotiation successful
0807 03:11:58.071314 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":269,"threads_started":1}
I20260807 03:11:58.073024 32361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:58.052772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:51993)
0807 03:11:58.053441 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:58.053484 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:58.055181 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:58.055894 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:58.055903 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:58.056365 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:58.057002 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:58.057020 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:58.058202 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:58.058208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:58.059881 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:58.059898 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:58.060664 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:58.060674 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:58.061024 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:58.061734 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:58.061766 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:58.064342 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:58.068660 (+ 4318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:58.068666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:58.068676 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:58.069088 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:58.069552 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:58.069556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:58.069558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:58.069728 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:58.070442 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:58.070447 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:58.070737 (+ 290us) client_negotiation.cc:769] Sending connection context
0807 03:11:58.070945 (+ 208us) client_negotiation.cc:240] Negotiation successful
0807 03:11:58.071494 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":289,"threads_started":1}
I20260807 03:11:58.074594 32358 heartbeater.cc:347] Connected to a master server at 127.17.31.62:35575
I20260807 03:11:58.074908 32358 heartbeater.cc:464] Registering TS with master...
I20260807 03:11:58.075591 32358 heartbeater.cc:511] Master 127.17.31.62:35575 requested a full tablet report, sending...
I20260807 03:11:58.077569 32160 ts_manager.cc:194] Registered new tserver with Master: fbc8f9b4702f43ae934d6eae53ac2bae (127.17.31.1:45557)
I20260807 03:11:58.079037 32160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:51993
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:11:58.115754 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35575
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:11:58.460057 32364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:11:58.460353 32364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:11:58.460525 32364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:11:58.467720 32364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:11:58.467867 32364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:11:58.468037 32364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:11:58.477607 32364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:11:58.478931 32364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:11:58.480261 32364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:11:58.493582 32369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:58.493583 32370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:58.493947 32372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:58.494835 32364 server_base.cc:1034] running on GCE node
I20260807 03:11:58.495396 32364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:11:58.496284 32364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:11:58.497509 32364 hybrid_clock.cc:648] HybridClock initialized: now 1786072318497474 us; error 43 us; skew 500 ppm
I20260807 03:11:58.497810 32364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:11:58 dist-test-slave-tm1g krb5kdc[32114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072318, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:58.500931 32364 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:11:58.502499 32364 webserver.cc:460] Webserver started at http://127.17.31.2:32981/ using document root <none> and password file <none>
I20260807 03:11:58.503161 32364 fs_manager.cc:359] Metadata directory not provided
I20260807 03:11:58.503314 32364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:11:58.503618 32364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:11:58.506471 32364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "606f50ac7e8f496984dc801745d52cad"
format_stamp: "Formatted at 2026-08-07 03:11:58 on dist-test-slave-tm1g"
server_key: "d1258780a00eabda26932eb54ad3606d"
server_key_iv: "9103e34ef27169da7365518cfe38de1f"
server_key_version: "encryptionkey@0"
I20260807 03:11:58.507277 32364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "606f50ac7e8f496984dc801745d52cad"
format_stamp: "Formatted at 2026-08-07 03:11:58 on dist-test-slave-tm1g"
server_key: "d1258780a00eabda26932eb54ad3606d"
server_key_iv: "9103e34ef27169da7365518cfe38de1f"
server_key_version: "encryptionkey@0"
I20260807 03:11:58.512395 32364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260807 03:11:58.515693 32379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:58.516938 32364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 03:11:58.517129 32364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "606f50ac7e8f496984dc801745d52cad"
format_stamp: "Formatted at 2026-08-07 03:11:58 on dist-test-slave-tm1g"
server_key: "d1258780a00eabda26932eb54ad3606d"
server_key_iv: "9103e34ef27169da7365518cfe38de1f"
server_key_version: "encryptionkey@0"
I20260807 03:11:58.517323 32364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:11:58.531852 32364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:11:58.535458 32364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:11:58.535732 32364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:11:58.536612 32364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:11:58.537879 32364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:11:58.537983 32364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:58.538136 32364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:11:58.538215 32364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:58.572358 32364 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:37093
I20260807 03:11:58.572413 32498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:37093 every 8 connection(s)
I20260807 03:11:58.573599 32364 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:11:58.580796 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 32364
I20260807 03:11:58.580999 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:11:58.581589 17532 external_mini_cluster.cc:1428] Setting key fb0fadaa8a2481f00cb9049f60f94a47
Aug 07 03:11:58 dist-test-slave-tm1g krb5kdc[32114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072318, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:11:58.593937 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:58.576212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:32903 (local address 127.17.31.62:35575)
0807 03:11:58.576656 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:58.576663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:58.577678 (+ 1015us) server_negotiation.cc:407] Connection header received
0807 03:11:58.579529 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:58.579533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:58.579600 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:58.579697 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:58.581336 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:58.582396 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:58.584153 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:58.584688 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:58.584824 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:58.588227 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:58.588232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:58.588235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:58.588265 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:58.590776 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:58.591460 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:58.591467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:58.591471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:58.591598 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:58.592077 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:58.592083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:58.592086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:58.592420 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:58.592594 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:58.593198 (+ 604us) server_negotiation.cc:299] Negotiation successful
0807 03:11:58.593457 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":206,"threads_started":1}
I20260807 03:11:58.594552 32502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:58.576811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:32903)
0807 03:11:58.577520 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:58.577563 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:58.579251 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:58.579892 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:58.579900 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:58.580511 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:58.581165 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:58.581185 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:58.582597 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:58.582602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:58.584015 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:58.584027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:58.584911 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:58.584919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:58.585145 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:58.585844 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:58.585866 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:58.588003 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:58.590919 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:58.590927 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:58.590941 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:58.591331 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:58.591730 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:58.591735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:58.591738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:58.591964 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:58.592564 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:58.592570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:58.592847 (+ 277us) client_negotiation.cc:769] Sending connection context
0807 03:11:58.593069 (+ 222us) client_negotiation.cc:240] Negotiation successful
0807 03:11:58.593421 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":276,"threads_started":1}
I20260807 03:11:58.595953 32499 heartbeater.cc:347] Connected to a master server at 127.17.31.62:35575
I20260807 03:11:58.596246 32499 heartbeater.cc:464] Registering TS with master...
I20260807 03:11:58.596951 32499 heartbeater.cc:511] Master 127.17.31.62:35575 requested a full tablet report, sending...
I20260807 03:11:58.598439 32160 ts_manager.cc:194] Registered new tserver with Master: 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093)
I20260807 03:11:58.599251 32160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:32903
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:11:58.635677 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35575
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:11:58.964993 32505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:11:58.965276 32505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:11:58.965478 32505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:11:58.972663 32505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:11:58.972802 32505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:11:58.972968 32505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:11:58.982486 32505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:11:58.983736 32505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:11:58.985009 32505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:11:58.997947 32513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:58.997992 32511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:11:58.998005 32510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:11:58.999158 32505 server_base.cc:1034] running on GCE node
I20260807 03:11:58.999749 32505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:11:59.000602 32505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:11:59.001825 32505 hybrid_clock.cc:648] HybridClock initialized: now 1786072319001801 us; error 43 us; skew 500 ppm
I20260807 03:11:59.002146 32505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:11:58 dist-test-slave-tm1g krb5kdc[32114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072318, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:11:59.005122 32505 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:11:59.006513 32505 webserver.cc:460] Webserver started at http://127.17.31.3:44153/ using document root <none> and password file <none>
I20260807 03:11:59.007180 32505 fs_manager.cc:359] Metadata directory not provided
I20260807 03:11:59.007321 32505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:11:59.007701 32505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:11:59.010797 32505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "353ad0f0cbb2421d8478072cfd8a1059"
format_stamp: "Formatted at 2026-08-07 03:11:59 on dist-test-slave-tm1g"
server_key: "0470401164f52968b4df155f9bcf47f3"
server_key_iv: "fa548749cd73c00df67747cff5fced8c"
server_key_version: "encryptionkey@0"
I20260807 03:11:59.011574 32505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "353ad0f0cbb2421d8478072cfd8a1059"
format_stamp: "Formatted at 2026-08-07 03:11:59 on dist-test-slave-tm1g"
server_key: "0470401164f52968b4df155f9bcf47f3"
server_key_iv: "fa548749cd73c00df67747cff5fced8c"
server_key_version: "encryptionkey@0"
I20260807 03:11:59.016460 32505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 03:11:59.019695 32520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:59.020937 32505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 03:11:59.021116 32505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "353ad0f0cbb2421d8478072cfd8a1059"
format_stamp: "Formatted at 2026-08-07 03:11:59 on dist-test-slave-tm1g"
server_key: "0470401164f52968b4df155f9bcf47f3"
server_key_iv: "fa548749cd73c00df67747cff5fced8c"
server_key_version: "encryptionkey@0"
I20260807 03:11:59.021353 32505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:11:59.056075 32505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:11:59.059834 32505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:11:59.060125 32505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:11:59.061070 32505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:11:59.062386 32505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:11:59.062486 32505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:59.062660 32505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:11:59.062752 32505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:11:59.081781 32358 heartbeater.cc:503] Master 127.17.31.62:35575 was elected leader, sending a full tablet report...
I20260807 03:11:59.104748 32505 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:34389
I20260807 03:11:59.104821 32636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:34389 every 8 connection(s)
I20260807 03:11:59.106022 32505 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:11:59.108610 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 32505
I20260807 03:11:59.108819 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:11:59.109301 17532 external_mini_cluster.cc:1428] Setting key 2e5a6a3b4edf03429ef53f75b1e56dd9
Aug 07 03:11:59 dist-test-slave-tm1g krb5kdc[32114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072318, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:11:59.126024 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.108559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:54471 (local address 127.17.31.62:35575)
0807 03:11:59.109130 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:59.109136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:59.110137 (+ 1001us) server_negotiation.cc:407] Connection header received
0807 03:11:59.112330 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:59.112336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:59.112419 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:59.112540 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:59.114018 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.114885 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.116330 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.116713 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.116803 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.119951 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:59.119955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:59.119958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:59.119987 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:59.122714 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:59.123590 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:59.123596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:59.123600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:59.123732 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:59.124163 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:59.124168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:59.124171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:59.124591 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:59.124753 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:59.125498 (+ 745us) server_negotiation.cc:299] Negotiation successful
0807 03:11:59.125684 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":265,"threads_started":1}
I20260807 03:11:59.126897 32640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.109145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:54471)
0807 03:11:59.109960 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:59.110020 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:59.112046 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:59.112823 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:59.112831 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:59.113293 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:11:59.113861 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:59.113878 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.115015 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.115019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:59.116216 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:59.116227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.116893 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.116898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:59.117080 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.117672 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:11:59.117697 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:11:59.119784 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:11:59.122894 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:59.122902 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:59.122914 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:59.123465 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:59.123844 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:11:59.123849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:11:59.123852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:11:59.124028 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:11:59.124726 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:11:59.124736 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:11:59.125129 (+ 393us) client_negotiation.cc:769] Sending connection context
0807 03:11:59.125388 (+ 259us) client_negotiation.cc:240] Negotiation successful
0807 03:11:59.125726 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":321,"threads_started":1}
I20260807 03:11:59.128139 32637 heartbeater.cc:347] Connected to a master server at 127.17.31.62:35575
I20260807 03:11:59.128422 32637 heartbeater.cc:464] Registering TS with master...
I20260807 03:11:59.129118 32637 heartbeater.cc:511] Master 127.17.31.62:35575 requested a full tablet report, sending...
I20260807 03:11:59.130486 32160 ts_manager.cc:194] Registered new tserver with Master: 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389)
I20260807 03:11:59.131301 32160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:54471
I20260807 03:11:59.136968 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:11:59.154376 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.141583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33226 (local address 127.17.31.62:35575)
0807 03:11:59.141735 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:59.141738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:59.142349 (+ 611us) server_negotiation.cc:407] Connection header received
0807 03:11:59.142528 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:59.142532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:59.142587 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:59.142657 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:59.144025 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.144729 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.146013 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.146467 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.146568 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.148758 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:59.148763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:59.148766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:59.148793 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:59.151629 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:59.152363 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:59.152375 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:59.152378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:59.152499 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:59.152892 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:59.152897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:59.152900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:59.153187 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:59.153444 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:59.153817 (+ 373us) server_negotiation.cc:299] Negotiation successful
0807 03:11:59.154030 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 03:11:59.158072 32160 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:11:59.164671 32160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:11:59.186539 32651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.176080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.17.31.2:37093)
0807 03:11:59.177063 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:59.177071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:59.177563 (+ 492us) server_negotiation.cc:407] Connection header received
0807 03:11:59.177861 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:59.177871 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:59.178168 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:59.178413 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:11:59.179334 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.180932 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.184134 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.185185 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.185330 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.185531 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:59.186015 (+ 484us) server_negotiation.cc:299] Negotiation successful
0807 03:11:59.186189 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":663,"threads_started":1}
I20260807 03:11:59.186587 32650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.176007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:37093 (local address 127.0.0.1:34726)
0807 03:11:59.177403 (+ 1396us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:59.177451 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:59.177622 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:59.178688 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:59.178699 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:59.178735 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:11:59.179116 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:59.179130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.181080 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.181088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:59.183966 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:59.183986 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.185421 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.185428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:59.185741 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.185799 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 03:11:59.185937 (+ 138us) client_negotiation.cc:240] Negotiation successful
0807 03:11:59.186120 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1184,"mutex_wait_us":41,"thread_start_us":1053,"threads_started":1}
I20260807 03:11:59.189064 32654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.177773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37782)
0807 03:11:59.178429 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:59.178451 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:59.178559 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:59.180158 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:59.180163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:59.180187 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:11:59.180612 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:59.180623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.182794 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.182798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:59.185116 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:59.185131 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.188238 (+ 3107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.188245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:59.188421 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.188462 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 03:11:59.188564 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 03:11:59.188691 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":1,"thread_start_us":308,"threads_started":1}
I20260807 03:11:59.189133 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.178107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37782 (local address 127.17.31.1:45557)
0807 03:11:59.179100 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:59.179108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:59.179135 (+ 27us) server_negotiation.cc:407] Connection header received
0807 03:11:59.179737 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:59.179746 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:59.179959 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:59.180782 (+ 823us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:11:59.181024 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.182587 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.185261 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.188011 (+ 2750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.188160 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.188352 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:59.188623 (+ 271us) server_negotiation.cc:299] Negotiation successful
0807 03:11:59.188788 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":667,"threads_started":1}
I20260807 03:11:59.190021 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.179265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:34389 (local address 127.0.0.1:37458)
0807 03:11:59.181575 (+ 2310us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:59.181598 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:59.181754 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:59.182380 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:59.182385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:59.182407 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:11:59.182926 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:59.182939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.185020 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.185025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:59.187130 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:59.187166 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.189277 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.189285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:59.189448 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.189470 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 03:11:59.189575 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 03:11:59.189692 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2179,"thread_start_us":1999,"threads_started":1}
I20260807 03:11:59.190128 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.178866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.17.31.3:34389)
0807 03:11:59.179576 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:59.179587 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:59.181703 (+ 2116us) server_negotiation.cc:407] Connection header received
0807 03:11:59.181948 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:59.181957 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:59.182154 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:59.182396 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:11:59.183253 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.184868 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.187338 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.188786 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.188916 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.189133 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:59.189626 (+ 493us) server_negotiation.cc:299] Negotiation successful
0807 03:11:59.189777 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":503,"threads_started":1}
I20260807 03:11:59.190791 32427 tablet_service.cc:1467] Processing CreateTablet for tablet 7b111ca6ba494300bca04cda9574f04c (DEFAULT_TABLE table=test-table [id=2b01fdb918f84f0ba96ac00ad3be1596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:11:59.192130 32427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b111ca6ba494300bca04cda9574f04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:11:59.192613 32287 tablet_service.cc:1467] Processing CreateTablet for tablet 7b111ca6ba494300bca04cda9574f04c (DEFAULT_TABLE table=test-table [id=2b01fdb918f84f0ba96ac00ad3be1596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:11:59.193902 32287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b111ca6ba494300bca04cda9574f04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:11:59.196710 32568 tablet_service.cc:1467] Processing CreateTablet for tablet 7b111ca6ba494300bca04cda9574f04c (DEFAULT_TABLE table=test-table [id=2b01fdb918f84f0ba96ac00ad3be1596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:11:59.197963 32568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b111ca6ba494300bca04cda9574f04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:11:59.204211 32658 tablet_bootstrap.cc:492] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae: Bootstrap starting.
I20260807 03:11:59.207723 32658 tablet_bootstrap.cc:654] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae: Neither blocks nor log segments found. Creating new log.
I20260807 03:11:59.208693 32658 log.cc:824] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae: Log is configured to *not* fsync() on all Append() calls
I20260807 03:11:59.209594 32659 tablet_bootstrap.cc:492] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad: Bootstrap starting.
I20260807 03:11:59.212385 32661 tablet_bootstrap.cc:492] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059: Bootstrap starting.
I20260807 03:11:59.212882 32658 tablet_bootstrap.cc:492] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae: No bootstrap required, opened a new log
I20260807 03:11:59.213150 32659 tablet_bootstrap.cc:654] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad: Neither blocks nor log segments found. Creating new log.
I20260807 03:11:59.213207 32658 ts_tablet_manager.cc:1397] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260807 03:11:59.214425 32659 log.cc:824] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad: Log is configured to *not* fsync() on all Append() calls
I20260807 03:11:59.215643 32661 tablet_bootstrap.cc:654] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059: Neither blocks nor log segments found. Creating new log.
I20260807 03:11:59.216570 32661 log.cc:824] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059: Log is configured to *not* fsync() on all Append() calls
I20260807 03:11:59.217481 32659 tablet_bootstrap.cc:492] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad: No bootstrap required, opened a new log
I20260807 03:11:59.217731 32659 ts_tablet_manager.cc:1397] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 03:11:59.219020 32661 tablet_bootstrap.cc:492] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059: No bootstrap required, opened a new log
I20260807 03:11:59.219264 32661 ts_tablet_manager.cc:1397] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260807 03:11:59.223701 32658 raft_consensus.cc:348] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } } peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } }
I20260807 03:11:59.224112 32658 raft_consensus.cc:374] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:11:59.224210 32658 raft_consensus.cc:729] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc8f9b4702f43ae934d6eae53ac2bae, State: Initialized, Role: FOLLOWER
I20260807 03:11:59.224951 32658 consensus_queue.cc:260] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } } peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } }
I20260807 03:11:59.227116 32661 raft_consensus.cc:348] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } } peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } }
I20260807 03:11:59.227541 32661 raft_consensus.cc:374] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:11:59.227690 32661 raft_consensus.cc:729] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353ad0f0cbb2421d8478072cfd8a1059, State: Initialized, Role: FOLLOWER
I20260807 03:11:59.228072 32658 ts_tablet_manager.cc:1428] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
I20260807 03:11:59.228154 32659 raft_consensus.cc:348] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } } peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } }
I20260807 03:11:59.228407 32661 consensus_queue.cc:260] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } } peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } }
I20260807 03:11:59.228569 32659 raft_consensus.cc:374] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:11:59.228694 32659 raft_consensus.cc:729] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606f50ac7e8f496984dc801745d52cad, State: Initialized, Role: FOLLOWER
I20260807 03:11:59.229511 32637 heartbeater.cc:503] Master 127.17.31.62:35575 was elected leader, sending a full tablet report...
I20260807 03:11:59.229434 32659 consensus_queue.cc:260] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } } peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } }
I20260807 03:11:59.230058 32661 ts_tablet_manager.cc:1428] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
I20260807 03:11:59.231590 32499 heartbeater.cc:503] Master 127.17.31.62:35575 was elected leader, sending a full tablet report...
I20260807 03:11:59.232142 32659 ts_tablet_manager.cc:1428] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260807 03:11:59.246984 32664 raft_consensus.cc:482] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:11:59.247186 32664 raft_consensus.cc:504] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } } peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } }
I20260807 03:11:59.247526 32665 raft_consensus.cc:482] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:11:59.247764 32665 raft_consensus.cc:504] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } } peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } }
I20260807 03:11:59.248893 32664 leader_election.cc:288] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093), 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389)
I20260807 03:11:59.249338 32665 leader_election.cc:288] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093), fbc8f9b4702f43ae934d6eae53ac2bae (127.17.31.1:45557)
I20260807 03:11:59.258580 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.250500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:57851 (local address 127.17.31.1:45557)
0807 03:11:59.250711 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:59.250719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:59.251477 (+ 758us) server_negotiation.cc:407] Connection header received
0807 03:11:59.251737 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:59.251742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:59.251842 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:59.251907 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:11:59.253156 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.254279 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.256703 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.257512 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.257672 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.257767 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:59.258141 (+ 374us) server_negotiation.cc:299] Negotiation successful
0807 03:11:59.258274 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":32}
I20260807 03:11:59.258908 32640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.249673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:37093 (local address 127.17.31.3:34527)
0807 03:11:59.249877 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:59.249897 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:59.250019 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:59.252122 (+ 2103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:59.252126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:59.252174 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:11:59.252413 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:59.252427 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.254193 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.254200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:59.256365 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:59.256379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.258127 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.258134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:59.258347 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.258386 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 03:11:59.258453 (+ 67us) client_negotiation.cc:240] Negotiation successful
0807 03:11:59.258581 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":20,"spinlock_wait_cycles":24704}
I20260807 03:11:59.259526 32669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.250075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.17.31.3:57851)
0807 03:11:59.251385 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:59.251405 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:59.251578 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:59.252122 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:59.252126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:59.252175 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:11:59.252952 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:59.252966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.254517 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.254521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:59.256579 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:59.256590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.257732 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.257738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:59.257959 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.257997 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 03:11:59.258964 (+ 967us) client_negotiation.cc:240] Negotiation successful
0807 03:11:59.259102 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1158,"mutex_wait_us":93,"spinlock_wait_cycles":4096,"thread_start_us":991,"threads_started":1}
I20260807 03:11:59.259718 32668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.250130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:34389 (local address 127.17.31.1:54685)
0807 03:11:59.251008 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:59.251031 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:59.251195 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:59.251875 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:59.251892 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:59.251917 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:11:59.252447 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:59.252458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.253942 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.253946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:59.256656 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:59.256671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.258463 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.258470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:59.258643 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.258677 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 03:11:59.259163 (+ 486us) client_negotiation.cc:240] Negotiation successful
0807 03:11:59.259307 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":175,"thread_start_us":591,"threads_started":1}
I20260807 03:11:59.260049 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.251328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:54685 (local address 127.17.31.3:34389)
0807 03:11:59.251500 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:59.251507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:59.251526 (+ 19us) server_negotiation.cc:407] Connection header received
0807 03:11:59.251584 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:59.251592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:59.251657 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:59.251738 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:11:59.252634 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.253751 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.256829 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.258062 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.258177 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.258272 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:59.258841 (+ 569us) server_negotiation.cc:299] Negotiation successful
0807 03:11:59.258991 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":60}
I20260807 03:11:59.260046 32670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.250515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:34527 (local address 127.17.31.2:37093)
0807 03:11:59.251823 (+ 1308us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:59.251829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:59.251850 (+ 21us) server_negotiation.cc:407] Connection header received
0807 03:11:59.251906 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:59.251911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:59.251977 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:59.252320 (+ 343us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:11:59.252692 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.254030 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.256583 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.257946 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.259391 (+ 1445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.259524 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:59.259573 (+ 49us) server_negotiation.cc:299] Negotiation successful
0807 03:11:59.259709 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1189,"thread_start_us":1412,"threads_started":1}
I20260807 03:11:59.260375 32651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.249285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:60905 (local address 127.17.31.2:37093)
0807 03:11:59.249500 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:59.249508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:59.250145 (+ 637us) server_negotiation.cc:407] Connection header received
0807 03:11:59.250514 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:59.250519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:59.250589 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:59.250670 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:11:59.251633 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.252837 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.257971 (+ 5134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.259168 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.259274 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.259360 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:59.259980 (+ 620us) server_negotiation.cc:299] Negotiation successful
0807 03:11:59.260119 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 03:11:59.260686 32310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b111ca6ba494300bca04cda9574f04c" candidate_uuid: "353ad0f0cbb2421d8478072cfd8a1059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" is_pre_election: true
I20260807 03:11:59.261051 32450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b111ca6ba494300bca04cda9574f04c" candidate_uuid: "353ad0f0cbb2421d8478072cfd8a1059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606f50ac7e8f496984dc801745d52cad" is_pre_election: true
I20260807 03:11:59.261157 32310 raft_consensus.cc:2454] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 353ad0f0cbb2421d8478072cfd8a1059 in term 0.
I20260807 03:11:59.261344 32667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.249307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:37093 (local address 127.17.31.1:60905)
0807 03:11:59.250050 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:59.250067 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:11:59.250280 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:11:59.250951 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:11:59.250959 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:11:59.251014 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:11:59.251448 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:59.251463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.254131 (+ 2668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.254137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:59.257805 (+ 3668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:11:59.257817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.259580 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.259587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:11:59.259781 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.259814 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 03:11:59.260758 (+ 944us) client_negotiation.cc:240] Negotiation successful
0807 03:11:59.260923 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":1311,"thread_start_us":400,"threads_started":1}
I20260807 03:11:59.261245 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b111ca6ba494300bca04cda9574f04c" candidate_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353ad0f0cbb2421d8478072cfd8a1059" is_pre_election: true
I20260807 03:11:59.261569 32450 raft_consensus.cc:2454] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 353ad0f0cbb2421d8478072cfd8a1059 in term 0.
I20260807 03:11:59.261647 32589 raft_consensus.cc:2454] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc8f9b4702f43ae934d6eae53ac2bae in term 0.
I20260807 03:11:59.262042 32448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b111ca6ba494300bca04cda9574f04c" candidate_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606f50ac7e8f496984dc801745d52cad" is_pre_election: true
I20260807 03:11:59.262095 32524 leader_election.cc:302] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353ad0f0cbb2421d8478072cfd8a1059, fbc8f9b4702f43ae934d6eae53ac2bae; no voters:
I20260807 03:11:59.262348 32448 raft_consensus.cc:2454] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc8f9b4702f43ae934d6eae53ac2bae in term 0.
I20260807 03:11:59.262527 32665 raft_consensus.cc:2781] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:11:59.262691 32665 raft_consensus.cc:482] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:11:59.262823 32665 raft_consensus.cc:3037] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:11:59.263103 32243 leader_election.cc:302] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606f50ac7e8f496984dc801745d52cad, fbc8f9b4702f43ae934d6eae53ac2bae; no voters:
I20260807 03:11:59.263468 32664 raft_consensus.cc:2781] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:11:59.263621 32664 raft_consensus.cc:482] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:11:59.263736 32664 raft_consensus.cc:3037] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:11:59.265151 32665 raft_consensus.cc:504] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } } peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } }
I20260807 03:11:59.265691 32665 leader_election.cc:288] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [CANDIDATE]: Term 1 election: Requested vote from peers 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093), fbc8f9b4702f43ae934d6eae53ac2bae (127.17.31.1:45557)
I20260807 03:11:59.266011 32664 raft_consensus.cc:504] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } } peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } }
I20260807 03:11:59.266219 32448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b111ca6ba494300bca04cda9574f04c" candidate_uuid: "353ad0f0cbb2421d8478072cfd8a1059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606f50ac7e8f496984dc801745d52cad"
I20260807 03:11:59.266209 32310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b111ca6ba494300bca04cda9574f04c" candidate_uuid: "353ad0f0cbb2421d8478072cfd8a1059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae"
I20260807 03:11:59.266422 32448 raft_consensus.cc:3037] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:11:59.266494 32310 raft_consensus.cc:2379] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 353ad0f0cbb2421d8478072cfd8a1059 in current term 1: Already voted for candidate fbc8f9b4702f43ae934d6eae53ac2bae in this term.
I20260807 03:11:59.266744 32664 leader_election.cc:288] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 1 election: Requested vote from peers 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093), 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389)
I20260807 03:11:59.267427 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b111ca6ba494300bca04cda9574f04c" candidate_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353ad0f0cbb2421d8478072cfd8a1059"
I20260807 03:11:59.267653 32589 raft_consensus.cc:2379] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fbc8f9b4702f43ae934d6eae53ac2bae in current term 1: Already voted for candidate 353ad0f0cbb2421d8478072cfd8a1059 in this term.
I20260807 03:11:59.268100 32450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b111ca6ba494300bca04cda9574f04c" candidate_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606f50ac7e8f496984dc801745d52cad"
I20260807 03:11:59.268700 32243 leader_election.cc:302] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbc8f9b4702f43ae934d6eae53ac2bae; no voters: 353ad0f0cbb2421d8478072cfd8a1059, 606f50ac7e8f496984dc801745d52cad
I20260807 03:11:59.268975 32664 raft_consensus.cc:2726] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260807 03:11:59.269424 32448 raft_consensus.cc:2454] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 353ad0f0cbb2421d8478072cfd8a1059 in term 1.
I20260807 03:11:59.269764 32524 leader_election.cc:302] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 353ad0f0cbb2421d8478072cfd8a1059, 606f50ac7e8f496984dc801745d52cad; no voters: fbc8f9b4702f43ae934d6eae53ac2bae
I20260807 03:11:59.270052 32665 raft_consensus.cc:2781] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:11:59.270533 32665 raft_consensus.cc:686] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [term 1 LEADER]: Becoming Leader. State: Replica: 353ad0f0cbb2421d8478072cfd8a1059, State: Running, Role: LEADER
I20260807 03:11:59.271008 32665 consensus_queue.cc:237] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } } peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } }
I20260807 03:11:59.275822 32159 catalog_manager.cc:5495] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 reported cstate change: term changed from 0 to 1, leader changed from <none> to 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3). New cstate: current_term: 1 leader_uuid: "353ad0f0cbb2421d8478072cfd8a1059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } health_report { overall_health: HEALTHY } } }
W20260807 03:11:59.302232 32359 tablet.cc:2369] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:11:59.309785 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.304239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.17.31.3:34389)
0807 03:11:59.304426 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:59.304432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:59.304547 (+ 115us) server_negotiation.cc:407] Connection header received
0807 03:11:59.304784 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:59.304788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:59.304879 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:59.305026 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:11:59.305453 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.306190 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.307666 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.308056 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.308186 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.308695 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:11:59.309167 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:11:59.309323 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:59.309419 (+ 96us) server_negotiation.cc:299] Negotiation successful
0807 03:11:59.309499 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 03:11:59.317111 32448 raft_consensus.cc:1261] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad [term 1 FOLLOWER]: Refusing update from remote peer 353ad0f0cbb2421d8478072cfd8a1059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:11:59.317220 32310 raft_consensus.cc:1261] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Refusing update from remote peer 353ad0f0cbb2421d8478072cfd8a1059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:11:59.317968 32671 consensus_queue.cc:1035] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:11:59.318342 32665 consensus_queue.cc:1035] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 03:11:59.325699 32500 tablet.cc:2369] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:11:59.342648 32160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:11:59.342924 32160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:11:59.346431 32160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649: Sending DeleteTablet for 3 replicas of tablet 7b111ca6ba494300bca04cda9574f04c
I20260807 03:11:59.347545 32427 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b111ca6ba494300bca04cda9574f04c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:11:59 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:34726
I20260807 03:11:59.347703 32287 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b111ca6ba494300bca04cda9574f04c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:11:59 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:37782
I20260807 03:11:59.348443 32684 tablet_replica.cc:330] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad: stopping tablet replica
I20260807 03:11:59.348582 32568 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b111ca6ba494300bca04cda9574f04c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:11:59 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:37458
I20260807 03:11:59.348850 32684 raft_consensus.cc:2229] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:11:59.349332 32684 raft_consensus.cc:2258] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:11:59.349634 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 32223
I20260807 03:11:59.350982 32686 tablet_replica.cc:330] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059: stopping tablet replica
I20260807 03:11:59.351477 32684 ts_tablet_manager.cc:1905] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:11:59.351469 32686 raft_consensus.cc:2229] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:11:59.352046 32686 raft_consensus.cc:2258] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:11:59.354142 32686 ts_tablet_manager.cc:1905] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:11:59.359265 32684 ts_tablet_manager.cc:1918] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:11:59.359436 32684 log.cc:1196] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/7b111ca6ba494300bca04cda9574f04c
I20260807 03:11:59.359978 32684 ts_tablet_manager.cc:1939] T 7b111ca6ba494300bca04cda9574f04c P 606f50ac7e8f496984dc801745d52cad: Deleting consensus metadata
I20260807 03:11:59.361748 32146 catalog_manager.cc:4841] TS 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093): tablet 7b111ca6ba494300bca04cda9574f04c (table test-table [id=2b01fdb918f84f0ba96ac00ad3be1596]) successfully deleted
Aug 07 03:11:59 dist-test-slave-tm1g krb5kdc[32114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:11:59.365561 32686 ts_tablet_manager.cc:1918] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:11:59.365723 32686 log.cc:1196] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/7b111ca6ba494300bca04cda9574f04c
I20260807 03:11:59.366263 32686 ts_tablet_manager.cc:1939] T 7b111ca6ba494300bca04cda9574f04c P 353ad0f0cbb2421d8478072cfd8a1059: Deleting consensus metadata
I20260807 03:11:59.367908 32146 catalog_manager.cc:4841] TS 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389): tablet 7b111ca6ba494300bca04cda9574f04c (table test-table [id=2b01fdb918f84f0ba96ac00ad3be1596]) successfully deleted
I20260807 03:11:59.368247 32685 tablet_replica.cc:330] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae: stopping tablet replica
I20260807 03:11:59.368682 32685 raft_consensus.cc:2229] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:11:59.369222 32685 raft_consensus.cc:2258] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:11:59.371336 32685 ts_tablet_manager.cc:1905] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:11:59.373266 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.350815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.17.31.1:45557)
0807 03:11:59.350991 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:59.350997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:59.351135 (+ 138us) server_negotiation.cc:407] Connection header received
0807 03:11:59.351506 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:59.351515 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:59.351628 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:59.351715 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:59.354037 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.355252 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.357311 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.358003 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.358102 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.365623 (+ 7521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:59.365643 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:59.365666 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:59.365721 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:59.370226 (+ 4505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:59.371003 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:59.371012 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:59.371016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:59.371153 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:59.371567 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:59.371573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:59.371576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:59.371965 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:59.372284 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:59.372594 (+ 310us) server_negotiation.cc:299] Negotiation successful
0807 03:11:59.372817 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 03:11:59.373974 32352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:11:59.479241 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 32223
I20260807 03:11:59.499184 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 32364
W20260807 03:11:59.499166 32146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.31.1:45557 (error 108)
I20260807 03:11:59.501250 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.500779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37794)
0807 03:11:59.500946 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:59.501071 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260807 03:11:59.501502 32146 catalog_manager.cc:4584] TS fbc8f9b4702f43ae934d6eae53ac2bae (127.17.31.1:45557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b111ca6ba494300bca04cda9574f04c: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
Aug 07 03:11:59 dist-test-slave-tm1g krb5kdc[32114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:11:59.517306 32651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.500499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34742 (local address 127.17.31.2:37093)
0807 03:11:59.500709 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:59.500715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:59.500776 (+ 61us) server_negotiation.cc:407] Connection header received
0807 03:11:59.501025 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:59.501031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:59.501120 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:59.501236 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:59.502991 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.503722 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.505658 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.506035 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.506124 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.511475 (+ 5351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:59.511489 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:59.511504 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:59.511539 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:59.514532 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:59.515208 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:59.515223 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:59.515226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:59.515344 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:59.515714 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:59.515720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:59.515723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:59.516124 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:59.516449 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:59.516754 (+ 305us) server_negotiation.cc:299] Negotiation successful
0807 03:11:59.516977 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 03:11:59.518015 32494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:11:59.555301 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.554746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37800)
0807 03:11:59.554985 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:59.555107 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260807 03:11:59.556463 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.556062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37804)
0807 03:11:59.556222 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:59.556307 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 03:11:59.609035 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 32364
I20260807 03:11:59.617214 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.616580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37820)
0807 03:11:59.616868 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:59.616995 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260807 03:11:59.618480 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.618034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37822)
0807 03:11:59.618217 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:59.618331 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260807 03:11:59.629112 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 32505
Aug 07 03:11:59 dist-test-slave-tm1g krb5kdc[32114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:11:59.649566 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.629982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37468 (local address 127.17.31.3:34389)
0807 03:11:59.630187 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 03:11:59.630194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:11:59.630215 (+ 21us) server_negotiation.cc:407] Connection header received
0807 03:11:59.630287 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:11:59.630294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:11:59.630450 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:11:59.630586 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:11:59.631871 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.633067 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.635090 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:11:59.635811 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:11:59.635966 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:11:59.642186 (+ 6220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:11:59.642206 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:11:59.642233 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:11:59.642280 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:11:59.646635 (+ 4355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:59.647316 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:59.647325 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:59.647329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:59.647498 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:11:59.647866 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:11:59.647872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:11:59.647876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:11:59.648292 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:11:59.648640 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0807 03:11:59.648975 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 03:11:59.649211 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 03:11:59.650341 32632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:11:59.699756 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.699241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37824)
0807 03:11:59.699469 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:59.699579 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 03:11:59.700947 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.700504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37834)
0807 03:11:59.700731 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:59.700815 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260807 03:11:59.778436 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 32505
I20260807 03:11:59.800407 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 32127
I20260807 03:11:59.801501 32192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:11:59.912323 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 32127
I20260807 03:11:59.912403 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:11:59.911843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37840)
0807 03:11:59.912062 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 03:11:59.912181 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 03:11:59.934888 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:35575
--webserver_interface=127.17.31.62
--webserver_port=44067
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:35575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:12:00.264386 32693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:12:00.264658 32693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:12:00.264823 32693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:12:00.271778 32693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:12:00.271874 32693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:12:00.271971 32693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:12:00.272080 32693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:12:00.272161 32693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:12:00.281500 32693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:35575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:35575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=44067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:12:00.282729 32693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:12:00.283988 32693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:12:00.306913 32700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:00.306913 32703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:00.306921 32701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:12:00.307891 32693 server_base.cc:1034] running on GCE node
I20260807 03:12:00.308712 32693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:12:00.310237 32693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:12:00.311494 32693 hybrid_clock.cc:648] HybridClock initialized: now 1786072320311443 us; error 35 us; skew 500 ppm
I20260807 03:12:00.311906 32693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:12:00 dist-test-slave-tm1g krb5kdc[32114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072320, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:12:00.315356 32693 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:12:00.316990 32693 webserver.cc:460] Webserver started at http://127.17.31.62:44067/ using document root <none> and password file <none>
I20260807 03:12:00.317763 32693 fs_manager.cc:359] Metadata directory not provided
I20260807 03:12:00.317932 32693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:12:00.323791 32693 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260807 03:12:00.326725 32711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:00.328066 32693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260807 03:12:00.328258 32693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "d2f1edcbb028476eaf98f25d4ea22649"
format_stamp: "Formatted at 2026-08-07 03:11:57 on dist-test-slave-tm1g"
server_key: "e5ea7860541af4f8c260d7a32513e249"
server_key_iv: "45d7c91a37cabc3da23859209ff9886c"
server_key_version: "encryptionkey@0"
I20260807 03:12:00.329304 32693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:12:00.364295 32693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:12:00.368400 32693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:12:00.368691 32693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:12:00.387734 32693 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:35575
I20260807 03:12:00.387758 32764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:35575 every 8 connection(s)
I20260807 03:12:00.389340 32693 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:12:00.390926 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 32693
I20260807 03:12:00.391958 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:45557
--local_ip_for_outbound_sockets=127.17.31.1
--tserver_master_addrs=127.17.31.62:35575
--webserver_port=34937
--webserver_interface=127.17.31.1
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:12:00.399333 32765 sys_catalog.cc:263] Verifying existing consensus state
I20260807 03:12:00.403837 32765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649: Bootstrap starting.
I20260807 03:12:00.423013 32765 log.cc:824] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649: Log is configured to *not* fsync() on all Append() calls
I20260807 03:12:00.435333 32765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:12:00.435873 32765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649: Bootstrap complete.
I20260807 03:12:00.445057 32765 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } }
I20260807 03:12:00.445835 32765 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f1edcbb028476eaf98f25d4ea22649, State: Initialized, Role: FOLLOWER
I20260807 03:12:00.446539 32765 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } }
I20260807 03:12:00.446805 32765 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:12:00.446938 32765 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:12:00.447131 32765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:12:00.450899 32765 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } }
I20260807 03:12:00.451403 32765 leader_election.cc:302] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2f1edcbb028476eaf98f25d4ea22649; no voters:
I20260807 03:12:00.452055 32765 leader_election.cc:288] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 03:12:00.452189 301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:12:00.453460 301 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 2 LEADER]: Becoming Leader. State: Replica: d2f1edcbb028476eaf98f25d4ea22649, State: Running, Role: LEADER
I20260807 03:12:00.454037 301 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } }
I20260807 03:12:00.454363 32765 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:12:00.456573 303 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2f1edcbb028476eaf98f25d4ea22649. Latest consensus state: current_term: 2 leader_uuid: "d2f1edcbb028476eaf98f25d4ea22649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } } }
I20260807 03:12:00.456959 303 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [sys.catalog]: This master's current role is: LEADER
I20260807 03:12:00.456671 302 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d2f1edcbb028476eaf98f25d4ea22649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } } }
I20260807 03:12:00.457080 302 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [sys.catalog]: This master's current role is: LEADER
I20260807 03:12:00.461390 308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:12:00.468987 308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:12:00.470395 308 catalog_manager.cc:1259] Loaded cluster ID: 58655f54010242ab891eb566c481737a
I20260807 03:12:00.470541 308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:12:00.473681 308 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260807 03:12:00.492552 308 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260807 03:12:00.501549 308 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:12:00.503333 308 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649: Loaded TSK: 0
I20260807 03:12:00.504908 308 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 03:12:00.748245 32767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:12:00.748504 32767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:12:00.748660 32767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:12:00.755923 32767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:12:00.756045 32767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:12:00.756255 32767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:12:00.766566 32767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:45557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=34937
--tserver_master_addrs=127.17.31.62:35575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:12:00.767853 32767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:12:00.769162 32767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:12:00.783545 326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:00.783802 328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:00.783809 325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:12:00.784453 32767 server_base.cc:1034] running on GCE node
I20260807 03:12:00.785163 32767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:12:00.786288 32767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:12:00.787557 32767 hybrid_clock.cc:648] HybridClock initialized: now 1786072320787511 us; error 38 us; skew 500 ppm
I20260807 03:12:00.787915 32767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:12:00 dist-test-slave-tm1g krb5kdc[32114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072320, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:12:00.797042 32767 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:12:00.798724 32767 webserver.cc:460] Webserver started at http://127.17.31.1:34937/ using document root <none> and password file <none>
I20260807 03:12:00.799624 32767 fs_manager.cc:359] Metadata directory not provided
I20260807 03:12:00.799778 32767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:12:00.806664 32767 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 03:12:00.810214 335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:00.811775 32767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 03:12:00.811941 32767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "fbc8f9b4702f43ae934d6eae53ac2bae"
format_stamp: "Formatted at 2026-08-07 03:11:57 on dist-test-slave-tm1g"
server_key: "a9b07e2c577b875865616afd95643ebb"
server_key_iv: "5ca80a34344e7985807784a496a241ef"
server_key_version: "encryptionkey@0"
I20260807 03:12:00.813232 32767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:12:00.832919 32767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:12:00.837615 32767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:12:00.837888 32767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:12:00.839010 32767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:12:00.840952 343 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260807 03:12:00.846062 343 ts_tablet_manager.cc:1852] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260807 03:12:00.847151 343 ts_tablet_manager.cc:1905] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:12:00.853864 343 ts_tablet_manager.cc:1918] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:12:00.854025 343 log.cc:1196] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/7b111ca6ba494300bca04cda9574f04c
I20260807 03:12:00.854615 343 ts_tablet_manager.cc:1939] T 7b111ca6ba494300bca04cda9574f04c P fbc8f9b4702f43ae934d6eae53ac2bae: Deleting consensus metadata
I20260807 03:12:00.854947 32767 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260807 03:12:00.855090 32767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260807 03:12:00.855233 32767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:12:00.855336 32767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:00.888159 32767 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:45557
I20260807 03:12:00.888244 454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:45557 every 8 connection(s)
I20260807 03:12:00.889642 32767 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:12:00.900040 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 32767
I20260807 03:12:00.900880 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:37093
--local_ip_for_outbound_sockets=127.17.31.2
--tserver_master_addrs=127.17.31.62:35575
--webserver_port=32981
--webserver_interface=127.17.31.2
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 03:12:00 dist-test-slave-tm1g krb5kdc[32114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072320, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:12:00.915359 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:00.893192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:56877)
0807 03:12:00.894107 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:00.894161 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:00.896007 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:00.897372 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:00.897379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:00.897882 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:12:00.898489 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:00.898509 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:00.900254 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:00.900259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:00.902426 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:00.902445 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:00.903458 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:00.903467 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:00.903768 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:00.904506 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:12:00.904541 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:12:00.907554 (+ 3013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:12:00.911509 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:00.911516 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:00.911526 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:00.911927 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:00.912411 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:00.912414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:00.912416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:00.912575 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:00.913471 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:12:00.913477 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:12:00.913738 (+ 261us) client_negotiation.cc:769] Sending connection context
0807 03:12:00.913947 (+ 209us) client_negotiation.cc:240] Negotiation successful
0807 03:12:00.914385 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":396,"threads_started":1}
I20260807 03:12:00.915874 457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:00.892917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:56877 (local address 127.17.31.62:35575)
0807 03:12:00.893542 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:00.893553 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:00.894288 (+ 735us) server_negotiation.cc:407] Connection header received
0807 03:12:00.896574 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:00.896595 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:00.896932 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:00.897236 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:00.898711 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:00.900043 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:00.902569 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:00.903218 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:00.903347 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:00.907830 (+ 4483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:00.907838 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:00.907850 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:00.907894 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:00.911370 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:00.912074 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:00.912085 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:00.912095 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:00.912301 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:00.912689 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:00.912696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:00.912702 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:00.913339 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:00.913548 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:00.914219 (+ 671us) server_negotiation.cc:299] Negotiation successful
0807 03:12:00.914670 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":266,"threads_started":1}
I20260807 03:12:00.918610 455 heartbeater.cc:347] Connected to a master server at 127.17.31.62:35575
I20260807 03:12:00.918919 455 heartbeater.cc:464] Registering TS with master...
I20260807 03:12:00.919660 455 heartbeater.cc:511] Master 127.17.31.62:35575 requested a full tablet report, sending...
I20260807 03:12:00.921700 32730 ts_manager.cc:194] Registered new tserver with Master: fbc8f9b4702f43ae934d6eae53ac2bae (127.17.31.1:45557)
I20260807 03:12:00.923249 32730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:56877
W20260807 03:12:01.233754 459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:12:01.234009 459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:12:01.234155 459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:12:01.241140 459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:12:01.241267 459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:12:01.241451 459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:12:01.250595 459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:37093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=32981
--tserver_master_addrs=127.17.31.62:35575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:12:01.251772 459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:12:01.253057 459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:12:01.266332 464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:01.266294 465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:01.266251 468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:12:01.267117 459 server_base.cc:1034] running on GCE node
I20260807 03:12:01.267622 459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:12:01.268433 459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:12:01.269652 459 hybrid_clock.cc:648] HybridClock initialized: now 1786072321269612 us; error 53 us; skew 500 ppm
I20260807 03:12:01.269964 459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:12:01 dist-test-slave-tm1g krb5kdc[32114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072321, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:12:01.273005 459 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:12:01.274372 459 webserver.cc:460] Webserver started at http://127.17.31.2:32981/ using document root <none> and password file <none>
I20260807 03:12:01.275034 459 fs_manager.cc:359] Metadata directory not provided
I20260807 03:12:01.275193 459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:12:01.280194 459 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260807 03:12:01.282733 475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:01.283975 459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 03:12:01.284147 459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "606f50ac7e8f496984dc801745d52cad"
format_stamp: "Formatted at 2026-08-07 03:11:58 on dist-test-slave-tm1g"
server_key: "d1258780a00eabda26932eb54ad3606d"
server_key_iv: "9103e34ef27169da7365518cfe38de1f"
server_key_version: "encryptionkey@0"
I20260807 03:12:01.285094 459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:12:01.301003 459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:12:01.305147 459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:12:01.305428 459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:12:01.306401 459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:12:01.307627 459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:12:01.307746 459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:01.307876 459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:12:01.307974 459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:01.341110 459 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:37093
I20260807 03:12:01.341204 590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:37093 every 8 connection(s)
I20260807 03:12:01.342582 459 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:12:01.349406 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 459
I20260807 03:12:01.350149 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:34389
--local_ip_for_outbound_sockets=127.17.31.3
--tserver_master_addrs=127.17.31.62:35575
--webserver_port=44153
--webserver_interface=127.17.31.3
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 03:12:01 dist-test-slave-tm1g krb5kdc[32114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072321, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:12:01.364180 457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:01.344943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:55727 (local address 127.17.31.62:35575)
0807 03:12:01.345161 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:01.345168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:01.346287 (+ 1119us) server_negotiation.cc:407] Connection header received
0807 03:12:01.348018 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:01.348023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:01.348090 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:01.348177 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:01.349820 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.350647 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.353234 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.353630 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.353725 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:01.357919 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:01.357927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:01.357935 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:01.357979 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:01.360945 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:01.361821 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:01.361825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:01.361827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:01.361942 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:01.362447 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:01.362451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:01.362453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:01.362737 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:01.362844 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:01.363686 (+ 842us) server_negotiation.cc:299] Negotiation successful
0807 03:12:01.363918 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 03:12:01.365590 593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:01.345433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:55727)
0807 03:12:01.346129 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:01.346165 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:01.347787 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:01.348382 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:01.348390 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:01.348850 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:12:01.349655 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:01.349681 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.350813 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.350821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:01.352984 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:01.353002 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.353868 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.353877 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:01.354246 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:01.355024 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:12:01.355061 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:12:01.357721 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:12:01.361092 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:01.361101 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:01.361115 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:01.361708 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:01.362059 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:01.362066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:01.362073 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:01.362333 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:01.362850 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:12:01.362859 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:12:01.363270 (+ 411us) client_negotiation.cc:769] Sending connection context
0807 03:12:01.363566 (+ 296us) client_negotiation.cc:240] Negotiation successful
0807 03:12:01.364179 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":272,"threads_started":1}
I20260807 03:12:01.367041 591 heartbeater.cc:347] Connected to a master server at 127.17.31.62:35575
I20260807 03:12:01.367326 591 heartbeater.cc:464] Registering TS with master...
I20260807 03:12:01.368006 591 heartbeater.cc:511] Master 127.17.31.62:35575 requested a full tablet report, sending...
I20260807 03:12:01.369414 32730 ts_manager.cc:194] Registered new tserver with Master: 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093)
I20260807 03:12:01.370215 32730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:55727
W20260807 03:12:01.679266 594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:12:01.679559 594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:12:01.679728 594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:12:01.686642 594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:12:01.686803 594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:12:01.686995 594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:12:01.696305 594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:34389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=44153
--tserver_master_addrs=127.17.31.62:35575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:12:01.697446 594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:12:01.698746 594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:12:01.711251 599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:01.711299 602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:01.711330 600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:12:01.712354 594 server_base.cc:1034] running on GCE node
I20260807 03:12:01.712924 594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:12:01.713769 594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:12:01.714955 594 hybrid_clock.cc:648] HybridClock initialized: now 1786072321714922 us; error 24 us; skew 500 ppm
I20260807 03:12:01.715242 594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:12:01 dist-test-slave-tm1g krb5kdc[32114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072321, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:12:01.718156 594 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:12:01.719677 594 webserver.cc:460] Webserver started at http://127.17.31.3:44153/ using document root <none> and password file <none>
I20260807 03:12:01.720264 594 fs_manager.cc:359] Metadata directory not provided
I20260807 03:12:01.720396 594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:12:01.725322 594 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 03:12:01.727834 609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:01.729074 594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 03:12:01.729242 594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "353ad0f0cbb2421d8478072cfd8a1059"
format_stamp: "Formatted at 2026-08-07 03:11:59 on dist-test-slave-tm1g"
server_key: "0470401164f52968b4df155f9bcf47f3"
server_key_iv: "fa548749cd73c00df67747cff5fced8c"
server_key_version: "encryptionkey@0"
I20260807 03:12:01.730167 594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:12:01.756944 594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:12:01.760671 594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:12:01.760934 594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:12:01.761835 594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:12:01.763201 594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:12:01.763322 594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:01.763474 594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:12:01.763564 594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:01.796813 594 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:34389
I20260807 03:12:01.796896 725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:34389 every 8 connection(s)
I20260807 03:12:01.798195 594 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:12:01.799167 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 594
Aug 07 03:12:01 dist-test-slave-tm1g krb5kdc[32114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072321, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:12:01.816359 457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:01.800340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.17.31.62:35575)
0807 03:12:01.800568 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:01.800574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:01.801160 (+ 586us) server_negotiation.cc:407] Connection header received
0807 03:12:01.801423 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:01.801428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:01.801507 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:01.801620 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:01.803607 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.804434 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.806493 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.807208 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.807329 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:01.810483 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:01.810490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:01.810496 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:01.810545 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:01.813337 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:01.814132 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:01.814137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:01.814140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:01.814351 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:01.814733 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:01.814741 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:01.814746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:01.815132 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:01.815412 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:01.815689 (+ 277us) server_negotiation.cc:299] Negotiation successful
0807 03:12:01.815928 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"spinlock_wait_cycles":2944}
I20260807 03:12:01.820587 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:01.800962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:45771 (local address 127.17.31.62:35575)
0807 03:12:01.801448 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:01.801453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:01.803630 (+ 2177us) server_negotiation.cc:407] Connection header received
0807 03:12:01.804391 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:01.804399 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:01.804474 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:01.804577 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:01.806465 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.807596 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.810040 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.810705 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.810835 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:01.814124 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:01.814131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:01.814135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:01.814189 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:01.817701 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:01.818560 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:01.818564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:01.818566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:01.818671 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:01.819228 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:01.819231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:01.819234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:01.819466 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:01.819575 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:01.820135 (+ 560us) server_negotiation.cc:299] Negotiation successful
0807 03:12:01.820272 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"spinlock_wait_cycles":896,"thread_start_us":274,"threads_started":1}
I20260807 03:12:01.821470 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:01.801621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:45771)
0807 03:12:01.802484 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:01.802538 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:01.804155 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:01.804822 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:01.804832 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:01.805531 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:12:01.806247 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:01.806271 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.807760 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.807768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:01.809882 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:01.809896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.810904 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.810908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:01.811090 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:01.811651 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:12:01.811670 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:12:01.813872 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:12:01.817817 (+ 3945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:01.817822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:01.817832 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:01.818455 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:01.818858 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:01.818863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:01.818866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:01.819097 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:01.819571 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:12:01.819575 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:12:01.819834 (+ 259us) client_negotiation.cc:769] Sending connection context
0807 03:12:01.820046 (+ 212us) client_negotiation.cc:240] Negotiation successful
0807 03:12:01.820380 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":339,"threads_started":1}
I20260807 03:12:01.822734 726 heartbeater.cc:347] Connected to a master server at 127.17.31.62:35575
I20260807 03:12:01.823004 726 heartbeater.cc:464] Registering TS with master...
I20260807 03:12:01.823696 726 heartbeater.cc:511] Master 127.17.31.62:35575 requested a full tablet report, sending...
I20260807 03:12:01.824988 32730 ts_manager.cc:194] Registered new tserver with Master: 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389)
I20260807 03:12:01.825774 32730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:45771
I20260807 03:12:01.831071 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:12:01.837740 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:01.831864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33244 (local address 127.17.31.62:35575)
0807 03:12:01.832066 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:01.832073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:01.832500 (+ 427us) server_negotiation.cc:407] Connection header received
0807 03:12:01.832726 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:01.832732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:01.832863 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:01.833038 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:12:01.833448 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.834254 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.835710 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.836131 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.836288 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:01.836782 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:12:01.837202 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:12:01.837350 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:01.837443 (+ 93us) server_negotiation.cc:299] Negotiation successful
0807 03:12:01.837516 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 03:12:01.838618 32730 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:12:01.845461 32730 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:12:01.865787 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:01.855981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37850)
0807 03:12:01.856862 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:01.856899 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:01.857049 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:01.858313 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:01.858322 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:01.858405 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:12:01.858805 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:01.858819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.860706 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.860713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:01.862290 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:01.862302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.864780 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.864786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:01.864984 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:01.865065 (+ 81us) client_negotiation.cc:769] Sending connection context
0807 03:12:01.865184 (+ 119us) client_negotiation.cc:240] Negotiation successful
0807 03:12:01.865368 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"spinlock_wait_cycles":11648,"thread_start_us":349,"threads_started":1}
I20260807 03:12:01.866786 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:01.856198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37850 (local address 127.17.31.1:45557)
0807 03:12:01.856976 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:01.856984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:01.857013 (+ 29us) server_negotiation.cc:407] Connection header received
0807 03:12:01.857356 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:01.857368 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:01.857895 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:01.858079 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:12:01.859006 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.860545 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.862503 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.864164 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.864285 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:01.866135 (+ 1850us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:01.866211 (+ 76us) server_negotiation.cc:299] Negotiation successful
0807 03:12:01.866402 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":507,"threads_started":1}
I20260807 03:12:01.868185 739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:01.857461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.17.31.3:34389)
0807 03:12:01.858979 (+ 1518us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:01.858988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:01.859021 (+ 33us) server_negotiation.cc:407] Connection header received
0807 03:12:01.859127 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:01.859134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:01.859357 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:01.860339 (+ 982us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:12:01.861247 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.862907 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.865705 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.867096 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.867216 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:01.867420 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:01.867674 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 03:12:01.867837 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1355,"thread_start_us":1219,"threads_started":1}
I20260807 03:12:01.868700 386 tablet_service.cc:1467] Processing CreateTablet for tablet 0f68fb9f13b6488bac8595f545ba1619 (DEFAULT_TABLE table=test-table [id=19a106a31bc948cb8a952801d684b495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:12:01.869764 386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f68fb9f13b6488bac8595f545ba1619. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:12:01.870774 740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:01.858227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:37093 (local address 127.0.0.1:34758)
0807 03:12:01.859226 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:01.859247 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:01.859348 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:01.860448 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:01.860452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:01.860480 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:12:01.860968 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:01.860983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.863093 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.863097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:01.865267 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:01.865279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.867166 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.867175 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:01.867358 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:01.867409 (+ 51us) client_negotiation.cc:769] Sending connection context
0807 03:12:01.867472 (+ 63us) client_negotiation.cc:240] Negotiation successful
0807 03:12:01.867568 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":81,"thread_start_us":812,"threads_started":1}
I20260807 03:12:01.871037 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:01.857053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:34389 (local address 127.0.0.1:37484)
0807 03:12:01.858206 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:01.858228 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:01.858382 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:01.859558 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:01.859566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:01.859596 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:12:01.859986 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:01.859998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.863063 (+ 3065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.863068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:01.865549 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:01.865563 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.867312 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.867318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:01.867483 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:01.867524 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 03:12:01.867619 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 03:12:01.867740 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":52,"thread_start_us":851,"threads_started":1}
I20260807 03:12:01.871913 741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:01.858859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34758 (local address 127.17.31.2:37093)
0807 03:12:01.859853 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:01.859864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:01.859897 (+ 33us) server_negotiation.cc:407] Connection header received
0807 03:12:01.859992 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:01.859999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:01.860213 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:01.861164 (+ 951us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:12:01.861461 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.862907 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.865440 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:01.866927 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:01.871225 (+ 4298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:01.871404 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:01.871465 (+ 61us) server_negotiation.cc:299] Negotiation successful
0807 03:12:01.871617 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":699,"threads_started":1}
I20260807 03:12:01.874480 660 tablet_service.cc:1467] Processing CreateTablet for tablet 0f68fb9f13b6488bac8595f545ba1619 (DEFAULT_TABLE table=test-table [id=19a106a31bc948cb8a952801d684b495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:12:01.875079 524 tablet_service.cc:1467] Processing CreateTablet for tablet 0f68fb9f13b6488bac8595f545ba1619 (DEFAULT_TABLE table=test-table [id=19a106a31bc948cb8a952801d684b495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:12:01.876084 660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f68fb9f13b6488bac8595f545ba1619. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:12:01.876288 524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f68fb9f13b6488bac8595f545ba1619. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:12:01.878048 742 tablet_bootstrap.cc:492] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae: Bootstrap starting.
I20260807 03:12:01.881105 742 tablet_bootstrap.cc:654] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae: Neither blocks nor log segments found. Creating new log.
I20260807 03:12:01.882520 742 log.cc:824] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae: Log is configured to *not* fsync() on all Append() calls
I20260807 03:12:01.884927 742 tablet_bootstrap.cc:492] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae: No bootstrap required, opened a new log
I20260807 03:12:01.885191 742 ts_tablet_manager.cc:1397] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260807 03:12:01.891373 744 tablet_bootstrap.cc:492] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059: Bootstrap starting.
I20260807 03:12:01.894840 744 tablet_bootstrap.cc:654] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059: Neither blocks nor log segments found. Creating new log.
I20260807 03:12:01.895586 745 tablet_bootstrap.cc:492] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad: Bootstrap starting.
I20260807 03:12:01.896126 744 log.cc:824] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059: Log is configured to *not* fsync() on all Append() calls
I20260807 03:12:01.897557 742 raft_consensus.cc:348] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } }
I20260807 03:12:01.899099 744 tablet_bootstrap.cc:492] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059: No bootstrap required, opened a new log
I20260807 03:12:01.899139 742 raft_consensus.cc:374] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:12:01.899260 742 raft_consensus.cc:729] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc8f9b4702f43ae934d6eae53ac2bae, State: Initialized, Role: FOLLOWER
I20260807 03:12:01.899328 744 ts_tablet_manager.cc:1397] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260807 03:12:01.899570 745 tablet_bootstrap.cc:654] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad: Neither blocks nor log segments found. Creating new log.
I20260807 03:12:01.900013 742 consensus_queue.cc:260] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } }
I20260807 03:12:01.901068 745 log.cc:824] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad: Log is configured to *not* fsync() on all Append() calls
I20260807 03:12:01.909664 742 ts_tablet_manager.cc:1428] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae: Time spent starting tablet: real 0.024s user 0.008s sys 0.012s
I20260807 03:12:01.909446 744 raft_consensus.cc:348] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } }
I20260807 03:12:01.909868 744 raft_consensus.cc:374] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:12:01.909982 744 raft_consensus.cc:729] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353ad0f0cbb2421d8478072cfd8a1059, State: Initialized, Role: FOLLOWER
I20260807 03:12:01.910108 455 heartbeater.cc:503] Master 127.17.31.62:35575 was elected leader, sending a full tablet report...
I20260807 03:12:01.910636 744 consensus_queue.cc:260] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } }
I20260807 03:12:01.911705 726 heartbeater.cc:503] Master 127.17.31.62:35575 was elected leader, sending a full tablet report...
I20260807 03:12:01.912302 744 ts_tablet_manager.cc:1428] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260807 03:12:01.912999 745 tablet_bootstrap.cc:492] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad: No bootstrap required, opened a new log
I20260807 03:12:01.913264 745 ts_tablet_manager.cc:1397] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20260807 03:12:01.920971 745 raft_consensus.cc:348] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } }
I20260807 03:12:01.921402 745 raft_consensus.cc:374] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:12:01.921520 745 raft_consensus.cc:729] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606f50ac7e8f496984dc801745d52cad, State: Initialized, Role: FOLLOWER
I20260807 03:12:01.922097 745 consensus_queue.cc:260] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } }
I20260807 03:12:01.923105 591 heartbeater.cc:503] Master 127.17.31.62:35575 was elected leader, sending a full tablet report...
I20260807 03:12:01.923667 745 ts_tablet_manager.cc:1428] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
W20260807 03:12:02.050078 727 tablet.cc:2369] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:12:02.090817 747 raft_consensus.cc:482] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:12:02.091156 747 raft_consensus.cc:504] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } }
I20260807 03:12:02.092986 747 leader_election.cc:288] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389), 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093)
W20260807 03:12:02.094597 592 tablet.cc:2369] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:12:02.100492 739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:02.093303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:33037 (local address 127.17.31.3:34389)
0807 03:12:02.093531 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:02.093539 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:02.094033 (+ 494us) server_negotiation.cc:407] Connection header received
0807 03:12:02.094216 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:02.094221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:02.094327 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:02.094424 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:12:02.095158 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:02.096026 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:02.098388 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:02.099568 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:02.099693 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:02.099796 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:02.100115 (+ 319us) server_negotiation.cc:299] Negotiation successful
0807 03:12:02.100251 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 03:12:02.100646 751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:02.093231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:34389 (local address 127.17.31.1:33037)
0807 03:12:02.093907 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:02.093927 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:02.094065 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:02.094570 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:02.094576 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:02.094601 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:12:02.094989 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:02.095002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:02.096169 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:02.096172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:02.098192 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:02.098207 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:02.099796 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:02.099801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:02.099933 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:02.099958 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 03:12:02.100079 (+ 121us) client_negotiation.cc:240] Negotiation successful
0807 03:12:02.100224 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":338,"threads_started":1}
I20260807 03:12:02.100944 741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:02.094210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:53829 (local address 127.17.31.2:37093)
0807 03:12:02.094481 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:02.094486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:02.094758 (+ 272us) server_negotiation.cc:407] Connection header received
0807 03:12:02.095022 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:02.095028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:02.095114 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:02.095199 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:12:02.095831 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:02.096602 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:02.098745 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:02.099937 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:02.100060 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:02.100150 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:02.100549 (+ 399us) server_negotiation.cc:299] Negotiation successful
0807 03:12:02.100690 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 03:12:02.101140 752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:02.094031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:37093 (local address 127.17.31.1:53829)
0807 03:12:02.094655 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:02.094680 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:02.094869 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:02.095235 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:02.095242 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:02.095267 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:12:02.095667 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:02.095678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:02.096732 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:02.096742 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:02.098586 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:02.098614 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:02.100158 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:02.100164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:02.100333 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:02.100372 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 03:12:02.100494 (+ 122us) client_negotiation.cc:240] Negotiation successful
0807 03:12:02.100618 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":935,"threads_started":1}
I20260807 03:12:02.101719 680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f68fb9f13b6488bac8595f545ba1619" candidate_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353ad0f0cbb2421d8478072cfd8a1059" is_pre_election: true
I20260807 03:12:02.102067 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f68fb9f13b6488bac8595f545ba1619" candidate_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606f50ac7e8f496984dc801745d52cad" is_pre_election: true
I20260807 03:12:02.102211 680 raft_consensus.cc:2454] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc8f9b4702f43ae934d6eae53ac2bae in term 0.
I20260807 03:12:02.102541 544 raft_consensus.cc:2454] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbc8f9b4702f43ae934d6eae53ac2bae in term 0.
I20260807 03:12:02.102943 340 leader_election.cc:302] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353ad0f0cbb2421d8478072cfd8a1059, fbc8f9b4702f43ae934d6eae53ac2bae; no voters:
I20260807 03:12:02.103278 747 raft_consensus.cc:2781] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:12:02.103425 747 raft_consensus.cc:482] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:12:02.103526 747 raft_consensus.cc:3037] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:12:02.105657 747 raft_consensus.cc:504] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } }
I20260807 03:12:02.106202 747 leader_election.cc:288] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 1 election: Requested vote from peers 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389), 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093)
I20260807 03:12:02.106753 680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f68fb9f13b6488bac8595f545ba1619" candidate_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353ad0f0cbb2421d8478072cfd8a1059"
I20260807 03:12:02.106802 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f68fb9f13b6488bac8595f545ba1619" candidate_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606f50ac7e8f496984dc801745d52cad"
I20260807 03:12:02.106972 680 raft_consensus.cc:3037] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:12:02.106972 544 raft_consensus.cc:3037] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:12:02.110128 544 raft_consensus.cc:2454] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc8f9b4702f43ae934d6eae53ac2bae in term 1.
I20260807 03:12:02.110128 680 raft_consensus.cc:2454] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbc8f9b4702f43ae934d6eae53ac2bae in term 1.
I20260807 03:12:02.110572 340 leader_election.cc:302] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606f50ac7e8f496984dc801745d52cad, fbc8f9b4702f43ae934d6eae53ac2bae; no voters:
I20260807 03:12:02.110831 747 raft_consensus.cc:2781] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:12:02.111248 747 raft_consensus.cc:686] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 LEADER]: Becoming Leader. State: Replica: fbc8f9b4702f43ae934d6eae53ac2bae, State: Running, Role: LEADER
I20260807 03:12:02.111689 747 consensus_queue.cc:237] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } }
I20260807 03:12:02.116168 32730 catalog_manager.cc:5495] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae reported cstate change: term changed from 0 to 1, leader changed from <none> to fbc8f9b4702f43ae934d6eae53ac2bae (127.17.31.1). New cstate: current_term: 1 leader_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } health_report { overall_health: UNKNOWN } } }
W20260807 03:12:02.142632 456 tablet.cc:2369] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:12:02.163524 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:02.157896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37866 (local address 127.17.31.1:45557)
0807 03:12:02.158129 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:02.158135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:02.158154 (+ 19us) server_negotiation.cc:407] Connection header received
0807 03:12:02.158344 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:02.158350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:02.158443 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:02.158592 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:12:02.158975 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:02.159720 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:02.161273 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:02.161684 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:02.161828 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:02.162409 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:12:02.162938 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:12:02.163089 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:02.163181 (+ 92us) server_negotiation.cc:299] Negotiation successful
0807 03:12:02.163259 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 03:12:02.171458 544 raft_consensus.cc:1261] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad [term 1 FOLLOWER]: Refusing update from remote peer fbc8f9b4702f43ae934d6eae53ac2bae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:12:02.171456 680 raft_consensus.cc:1261] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059 [term 1 FOLLOWER]: Refusing update from remote peer fbc8f9b4702f43ae934d6eae53ac2bae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:12:02.172390 753 consensus_queue.cc:1035] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [LEADER]: Connected to new peer: Peer: permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:12:02.172756 747 consensus_queue.cc:1035] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [LEADER]: Connected to new peer: Peer: permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:12:02.184541 757 mvcc.cc:205] Tried to move back new op lower bound from 7315752231604293632 to 7315752231372947456. Current Snapshot: MvccSnapshot[applied={T|T < 7315752231604293632}]
I20260807 03:12:02.196621 32729 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:12:02.196885 32729 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:12:02.200511 32729 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649: Sending DeleteTablet for 3 replicas of tablet 0f68fb9f13b6488bac8595f545ba1619
I20260807 03:12:02.201603 386 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f68fb9f13b6488bac8595f545ba1619 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:12:02 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:37850
I20260807 03:12:02.201772 660 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f68fb9f13b6488bac8595f545ba1619 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:12:02 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:37484
I20260807 03:12:02.201910 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 32767
I20260807 03:12:02.202867 524 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f68fb9f13b6488bac8595f545ba1619 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:12:02 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:34758
I20260807 03:12:02.217448 768 tablet_replica.cc:330] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae: stopping tablet replica
I20260807 03:12:02.217985 768 raft_consensus.cc:2229] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:12:02.218792 768 raft_consensus.cc:2258] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:12:02.221357 768 ts_tablet_manager.cc:1905] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:12:02.222406 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:02.203427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.17.31.1:45557)
0807 03:12:02.203615 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:02.203620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:02.203640 (+ 20us) server_negotiation.cc:407] Connection header received
0807 03:12:02.203692 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:02.203698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:02.203778 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:02.203886 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:02.206053 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:02.207402 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:02.209765 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:02.210474 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:02.210614 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:02.214869 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:02.214880 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:02.214896 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:02.214947 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:02.219185 (+ 4238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:02.220035 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:02.220043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:02.220047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:02.220231 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:02.220629 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:02.220634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:02.220638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:02.221038 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:02.221332 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:02.221733 (+ 401us) server_negotiation.cc:299] Negotiation successful
0807 03:12:02.221949 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 03:12:02.223083 450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:12:02.224128 771 tablet_replica.cc:330] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad: stopping tablet replica
I20260807 03:12:02.224507 771 raft_consensus.cc:2229] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:12:02.224938 771 raft_consensus.cc:2258] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:12:02.226471 770 tablet_replica.cc:330] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059: stopping tablet replica
I20260807 03:12:02.226660 771 ts_tablet_manager.cc:1905] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:12:02.226965 770 raft_consensus.cc:2229] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:12:02.227535 770 raft_consensus.cc:2258] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:12:02.229960 770 ts_tablet_manager.cc:1905] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:12:02.232295 771 ts_tablet_manager.cc:1918] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:12:02.232458 771 log.cc:1196] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/0f68fb9f13b6488bac8595f545ba1619
I20260807 03:12:02.233121 771 ts_tablet_manager.cc:1939] T 0f68fb9f13b6488bac8595f545ba1619 P 606f50ac7e8f496984dc801745d52cad: Deleting consensus metadata
I20260807 03:12:02.235033 32716 catalog_manager.cc:4841] TS 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093): tablet 0f68fb9f13b6488bac8595f545ba1619 (table test-table [id=19a106a31bc948cb8a952801d684b495]) successfully deleted
I20260807 03:12:02.237368 770 ts_tablet_manager.cc:1918] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:12:02.237489 770 log.cc:1196] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/0f68fb9f13b6488bac8595f545ba1619
I20260807 03:12:02.237957 770 ts_tablet_manager.cc:1939] T 0f68fb9f13b6488bac8595f545ba1619 P 353ad0f0cbb2421d8478072cfd8a1059: Deleting consensus metadata
I20260807 03:12:02.239378 32716 catalog_manager.cc:4841] TS 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389): tablet 0f68fb9f13b6488bac8595f545ba1619 (table test-table [id=19a106a31bc948cb8a952801d684b495]) successfully deleted
I20260807 03:12:02.330904 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 32767
I20260807 03:12:02.354588 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 459
W20260807 03:12:02.354624 32716 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.31.1:45557 (error 108)
I20260807 03:12:02.358072 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:02.357634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37884)
0807 03:12:02.357834 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:02.357941 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260807 03:12:02.358547 32716 catalog_manager.cc:4584] TS fbc8f9b4702f43ae934d6eae53ac2bae (127.17.31.1:45557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f68fb9f13b6488bac8595f545ba1619: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
I20260807 03:12:02.371008 741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:02.355353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34764 (local address 127.17.31.2:37093)
0807 03:12:02.356256 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:02.356264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:02.356289 (+ 25us) server_negotiation.cc:407] Connection header received
0807 03:12:02.356362 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:02.356370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:02.356504 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:02.356630 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:02.358424 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:02.359379 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:02.361312 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:02.361725 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:02.361831 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:02.365271 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:02.365281 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:02.365308 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:02.365349 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:02.368334 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:02.369080 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:02.369086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:02.369088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:02.369203 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:02.369588 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:02.369591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:02.369593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:02.369974 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:02.370193 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:02.370511 (+ 318us) server_negotiation.cc:299] Negotiation successful
0807 03:12:02.370661 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713}
I20260807 03:12:02.371766 585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:12:02.411947 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:02.411518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37900)
0807 03:12:02.411717 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:02.411814 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 03:12:02.412901 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:02.412590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37916)
0807 03:12:02.412726 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:02.412794 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 03:12:02.464035 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 459
I20260807 03:12:02.473600 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:02.473003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37928)
0807 03:12:02.473247 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:02.473371 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260807 03:12:02.474787 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:02.474296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37942)
0807 03:12:02.474524 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:02.474642 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260807 03:12:02.484436 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 594
I20260807 03:12:02.497638 739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:02.485168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37486 (local address 127.17.31.3:34389)
0807 03:12:02.485355 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:02.485360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:02.485379 (+ 19us) server_negotiation.cc:407] Connection header received
0807 03:12:02.485495 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:02.485499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:02.485581 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:02.485663 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:02.487174 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:02.487890 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:02.489107 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:02.489495 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:02.489587 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:02.492122 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:02.492136 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:02.492153 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:02.492183 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:02.495197 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:02.495815 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:02.495821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:02.495823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:02.495958 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:02.496286 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:02.496289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:02.496291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:02.496638 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:02.496845 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:02.497087 (+ 242us) server_negotiation.cc:299] Negotiation successful
0807 03:12:02.497263 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 03:12:02.498281 721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:12:02.555799 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:02.555253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37958)
0807 03:12:02.555507 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:02.555617 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260807 03:12:02.556929 738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:02.556526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:45557 (local address 127.0.0.1:37962)
0807 03:12:02.556750 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:02.556829 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260807 03:12:02.586719 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 594
I20260807 03:12:02.605430 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 32693
I20260807 03:12:02.606688 32760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:12:02.720299 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 32693
I20260807 03:12:02.743842 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:35575
--webserver_interface=127.17.31.62
--webserver_port=44067
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:35575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:12:03.067425 777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:12:03.067688 777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:12:03.067852 777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:12:03.074856 777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:12:03.074944 777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:12:03.075042 777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:12:03.075145 777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:12:03.075204 777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:12:03.084381 777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:35575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:35575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=44067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:12:03.085589 777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:12:03.086935 777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:12:03.098487 782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:03.098580 783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:03.098822 785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:12:03.099623 777 server_base.cc:1034] running on GCE node
I20260807 03:12:03.100359 777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:12:03.101677 777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:12:03.102923 777 hybrid_clock.cc:648] HybridClock initialized: now 1786072323102877 us; error 32 us; skew 500 ppm
I20260807 03:12:03.103273 777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:12:03 dist-test-slave-tm1g krb5kdc[32114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072323, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:12:03.106469 777 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:12:03.107811 777 webserver.cc:460] Webserver started at http://127.17.31.62:44067/ using document root <none> and password file <none>
I20260807 03:12:03.108417 777 fs_manager.cc:359] Metadata directory not provided
I20260807 03:12:03.108518 777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:12:03.113602 777 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:12:03.116460 792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:03.117782 777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:12:03.117969 777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "d2f1edcbb028476eaf98f25d4ea22649"
format_stamp: "Formatted at 2026-08-07 03:11:57 on dist-test-slave-tm1g"
server_key: "e5ea7860541af4f8c260d7a32513e249"
server_key_iv: "45d7c91a37cabc3da23859209ff9886c"
server_key_version: "encryptionkey@0"
I20260807 03:12:03.118961 777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:12:03.139891 777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:12:03.143872 777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:12:03.144119 777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:12:03.162353 777 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:35575
I20260807 03:12:03.162427 845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:35575 every 8 connection(s)
I20260807 03:12:03.163785 777 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:12:03.167387 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 777
I20260807 03:12:03.168237 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:45557
--local_ip_for_outbound_sockets=127.17.31.1
--tserver_master_addrs=127.17.31.62:35575
--webserver_port=34937
--webserver_interface=127.17.31.1
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:12:03.170478 846 sys_catalog.cc:263] Verifying existing consensus state
I20260807 03:12:03.173499 846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649: Bootstrap starting.
I20260807 03:12:03.200627 846 log.cc:824] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649: Log is configured to *not* fsync() on all Append() calls
I20260807 03:12:03.221009 846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:12:03.221621 846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649: Bootstrap complete.
I20260807 03:12:03.235311 846 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } }
I20260807 03:12:03.236788 846 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f1edcbb028476eaf98f25d4ea22649, State: Initialized, Role: FOLLOWER
I20260807 03:12:03.237579 846 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } }
I20260807 03:12:03.237906 846 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:12:03.238082 846 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:12:03.238305 846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:12:03.243685 846 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } }
I20260807 03:12:03.244153 846 leader_election.cc:302] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2f1edcbb028476eaf98f25d4ea22649; no voters:
I20260807 03:12:03.244836 846 leader_election.cc:288] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 03:12:03.244980 850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:12:03.246182 850 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [term 3 LEADER]: Becoming Leader. State: Replica: d2f1edcbb028476eaf98f25d4ea22649, State: Running, Role: LEADER
I20260807 03:12:03.246747 850 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } }
I20260807 03:12:03.247094 846 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:12:03.249140 852 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2f1edcbb028476eaf98f25d4ea22649. Latest consensus state: current_term: 3 leader_uuid: "d2f1edcbb028476eaf98f25d4ea22649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } } }
I20260807 03:12:03.249673 851 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d2f1edcbb028476eaf98f25d4ea22649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f1edcbb028476eaf98f25d4ea22649" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35575 } } }
I20260807 03:12:03.250110 851 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [sys.catalog]: This master's current role is: LEADER
I20260807 03:12:03.254634 852 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f1edcbb028476eaf98f25d4ea22649 [sys.catalog]: This master's current role is: LEADER
I20260807 03:12:03.266934 858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:12:03.275866 858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:12:03.277248 858 catalog_manager.cc:1259] Loaded cluster ID: 58655f54010242ab891eb566c481737a
I20260807 03:12:03.277343 858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260807 03:12:03.279320 858 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3a866969ad google::LogMessage::Fail() at ??:0
@ 0x7f3a8669988a google::LogMessage::SendToLog() at ??:0
@ 0x7f3a866964de google::LogMessage::Flush() at ??:0
@ 0x7f3a86697a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3a9c0c9512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f3a9c0c6976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3a9c14e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f3a9c14e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3a892e2345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3a892e591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f3a892e5799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3a892c0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3a8c59a6db start_thread at ??:0
@ 0x7f3a8553061f clone at ??:0
W20260807 03:12:03.599432 848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:12:03.599690 848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:12:03.599856 848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:12:03.606896 848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:12:03.607025 848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:12:03.607215 848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:12:03.616300 848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:45557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=34937
--tserver_master_addrs=127.17.31.62:35575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:12:03.617444 848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:12:03.618786 848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:12:03.631458 891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:03.631729 893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:03.631497 889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:12:03.632148 848 server_base.cc:1034] running on GCE node
I20260807 03:12:03.632676 848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:12:03.633482 848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:12:03.634675 848 hybrid_clock.cc:648] HybridClock initialized: now 1786072323634639 us; error 32 us; skew 500 ppm
I20260807 03:12:03.634979 848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:12:03 dist-test-slave-tm1g krb5kdc[32114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072323, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:12:03.637995 848 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:12:03.639498 848 webserver.cc:460] Webserver started at http://127.17.31.1:34937/ using document root <none> and password file <none>
I20260807 03:12:03.640188 848 fs_manager.cc:359] Metadata directory not provided
I20260807 03:12:03.640336 848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:12:03.645380 848 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260807 03:12:03.651202 900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:03.652755 848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 03:12:03.652915 848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "fbc8f9b4702f43ae934d6eae53ac2bae"
format_stamp: "Formatted at 2026-08-07 03:11:57 on dist-test-slave-tm1g"
server_key: "a9b07e2c577b875865616afd95643ebb"
server_key_iv: "5ca80a34344e7985807784a496a241ef"
server_key_version: "encryptionkey@0"
I20260807 03:12:03.653865 848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:12:03.673629 848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:12:03.677198 848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:12:03.677452 848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:12:03.678377 848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:12:03.680071 907 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 03:12:03.685366 848 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 03:12:03.685508 848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260807 03:12:03.685644 848 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 03:12:03.688676 848 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 03:12:03.688750 848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260807 03:12:03.689177 907 tablet_bootstrap.cc:492] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae: Bootstrap starting.
I20260807 03:12:03.697734 907 log.cc:824] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae: Log is configured to *not* fsync() on all Append() calls
I20260807 03:12:03.712842 907 tablet_bootstrap.cc:492] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:12:03.713721 907 tablet_bootstrap.cc:492] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae: Bootstrap complete.
I20260807 03:12:03.714279 907 ts_tablet_manager.cc:1397] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.004s
I20260807 03:12:03.724792 907 raft_consensus.cc:348] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } }
I20260807 03:12:03.726271 907 raft_consensus.cc:729] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc8f9b4702f43ae934d6eae53ac2bae, State: Initialized, Role: FOLLOWER
I20260807 03:12:03.727010 907 consensus_queue.cc:260] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } }
I20260807 03:12:03.728492 907 ts_tablet_manager.cc:1428] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260807 03:12:03.731228 848 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:45557
I20260807 03:12:03.731300 1021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:45557 every 8 connection(s)
I20260807 03:12:03.732321 848 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:12:03.736522 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:03.734853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:55009)
0807 03:12:03.735573 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:03.735772 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":308,"threads_started":1}
W20260807 03:12:03.737118 1022 heartbeater.cc:644] Failed to heartbeat to 127.17.31.62:35575 (0 consecutive failures): Network error: Failed to ping master at 127.17.31.62:35575: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
I20260807 03:12:03.737939 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:03.737613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:58991)
0807 03:12:03.737782 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:03.737843 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260807 03:12:03.738214 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 848
I20260807 03:12:03.738821 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:03.738488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:49897)
0807 03:12:03.738653 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:03.738717 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260807 03:12:03.738910 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:37093
--local_ip_for_outbound_sockets=127.17.31.2
--tserver_master_addrs=127.17.31.62:35575
--webserver_port=32981
--webserver_interface=127.17.31.2
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:12:03.739108 1022 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 03:12:04.060186 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:12:04.060442 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:12:04.060532 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:12:04.067354 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:12:04.067448 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:12:04.067564 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:12:04.076704 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:37093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=32981
--tserver_master_addrs=127.17.31.62:35575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:12:04.077785 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:12:04.078989 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:12:04.091444 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:04.091566 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:04.091895 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:12:04.092383 1025 server_base.cc:1034] running on GCE node
I20260807 03:12:04.092944 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:12:04.093827 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:12:04.095041 1025 hybrid_clock.cc:648] HybridClock initialized: now 1786072324095016 us; error 79 us; skew 500 ppm
I20260807 03:12:04.095331 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:12:04 dist-test-slave-tm1g krb5kdc[32114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072324, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:12:04.098284 1025 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:12:04.099660 1025 webserver.cc:460] Webserver started at http://127.17.31.2:32981/ using document root <none> and password file <none>
I20260807 03:12:04.100312 1025 fs_manager.cc:359] Metadata directory not provided
I20260807 03:12:04.100462 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:12:04.105564 1025 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260807 03:12:04.108124 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:04.109479 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:12:04.109673 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "606f50ac7e8f496984dc801745d52cad"
format_stamp: "Formatted at 2026-08-07 03:11:58 on dist-test-slave-tm1g"
server_key: "d1258780a00eabda26932eb54ad3606d"
server_key_iv: "9103e34ef27169da7365518cfe38de1f"
server_key_version: "encryptionkey@0"
I20260807 03:12:04.110697 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:12:04.144490 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:12:04.148291 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:12:04.148587 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:12:04.149520 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:12:04.150861 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:12:04.150982 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:04.151122 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:12:04.151211 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:04.184281 1025 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:37093
I20260807 03:12:04.184394 1159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:37093 every 8 connection(s)
I20260807 03:12:04.185719 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:12:04.190416 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:04.188646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:56945)
0807 03:12:04.189349 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:04.189546 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":291,"threads_started":1}
W20260807 03:12:04.191059 1160 heartbeater.cc:644] Failed to heartbeat to 127.17.31.62:35575 (0 consecutive failures): Network error: Failed to ping master at 127.17.31.62:35575: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
I20260807 03:12:04.191774 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:04.191489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:41203)
0807 03:12:04.191620 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:04.191680 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 03:12:04.192595 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:04.192311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:53555)
0807 03:12:04.192424 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:04.192499 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260807 03:12:04.192843 1160 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 03:12:04.194476 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 1025
I20260807 03:12:04.195096 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:34389
--local_ip_for_outbound_sockets=127.17.31.3
--tserver_master_addrs=127.17.31.62:35575
--webserver_port=44153
--webserver_interface=127.17.31.3
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:12:04.521626 1163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:12:04.521873 1163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:12:04.521961 1163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:12:04.528977 1163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:12:04.529076 1163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:12:04.529181 1163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:12:04.538499 1163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:34389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=44153
--tserver_master_addrs=127.17.31.62:35575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:12:04.539708 1163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:12:04.540948 1163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:12:04.553687 1169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:04.553656 1171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:04.553772 1168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:12:04.554653 1163 server_base.cc:1034] running on GCE node
I20260807 03:12:04.555266 1163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:12:04.556216 1163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:12:04.557426 1163 hybrid_clock.cc:648] HybridClock initialized: now 1786072324557390 us; error 47 us; skew 500 ppm
I20260807 03:12:04.557720 1163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:12:04 dist-test-slave-tm1g krb5kdc[32114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072324, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:12:04.560794 1163 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:12:04.562148 1163 webserver.cc:460] Webserver started at http://127.17.31.3:44153/ using document root <none> and password file <none>
I20260807 03:12:04.562846 1163 fs_manager.cc:359] Metadata directory not provided
I20260807 03:12:04.562992 1163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:12:04.567989 1163 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 03:12:04.570525 1179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:04.571756 1163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:12:04.571943 1163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "353ad0f0cbb2421d8478072cfd8a1059"
format_stamp: "Formatted at 2026-08-07 03:11:59 on dist-test-slave-tm1g"
server_key: "0470401164f52968b4df155f9bcf47f3"
server_key_iv: "fa548749cd73c00df67747cff5fced8c"
server_key_version: "encryptionkey@0"
I20260807 03:12:04.572877 1163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:12:04.584201 1163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:12:04.587865 1163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:12:04.588145 1163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:12:04.589072 1163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:12:04.590355 1163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:12:04.590477 1163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:04.590601 1163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:12:04.590708 1163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:04.624289 1163 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:34389
I20260807 03:12:04.624366 1295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:34389 every 8 connection(s)
I20260807 03:12:04.625733 1163 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:12:04.630365 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:04.628615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:50013)
0807 03:12:04.629320 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:04.629511 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":283,"threads_started":1}
I20260807 03:12:04.630368 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 1163
W20260807 03:12:04.631070 1296 heartbeater.cc:644] Failed to heartbeat to 127.17.31.62:35575 (0 consecutive failures): Network error: Failed to ping master at 127.17.31.62:35575: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
I20260807 03:12:04.631904 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:04.631588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:54207)
0807 03:12:04.631729 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:04.631804 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 03:12:04.632822 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:04.632509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:40243)
0807 03:12:04.632638 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:04.632713 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 03:12:04.633109 1296 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 03:12:04.741209 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:04.739690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:45303)
0807 03:12:04.740687 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:04.740879 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":702,"threads_started":1}
I20260807 03:12:05.187808 1302 raft_consensus.cc:482] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:12:05.188158 1302 raft_consensus.cc:504] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } }
I20260807 03:12:05.189878 1302 leader_election.cc:288] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389), 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093)
I20260807 03:12:05.194450 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:05.193291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:52451)
0807 03:12:05.193865 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:05.194070 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":338,"threads_started":1}
Aug 07 03:12:05 dist-test-slave-tm1g krb5kdc[32114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072323, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
Aug 07 03:12:05 dist-test-slave-tm1g krb5kdc[32114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072323, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:12:05.214866 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:05.190086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:34389 (local address 127.17.31.1:60595)
0807 03:12:05.190282 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:05.190350 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:05.190888 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:05.194288 (+ 3400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:05.194337 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:05.195249 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:12:05.196400 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:05.196428 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:05.198745 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:05.198751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:05.201163 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:05.201179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:05.202585 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:05.202590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:05.203004 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:05.203870 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:12:05.203919 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:12:05.207419 (+ 3500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:12:05.211580 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:05.211584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:05.211590 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:05.212075 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:05.212532 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:05.212537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:05.212540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:05.212693 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:05.213586 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:12:05.213591 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:12:05.213814 (+ 223us) client_negotiation.cc:769] Sending connection context
0807 03:12:05.213966 (+ 152us) client_negotiation.cc:240] Negotiation successful
0807 03:12:05.214414 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":566,"spinlock_wait_cycles":15488}
I20260807 03:12:05.214927 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:05.190971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:34945 (local address 127.17.31.2:37093)
0807 03:12:05.191620 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:05.191633 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:05.191666 (+ 33us) server_negotiation.cc:407] Connection header received
0807 03:12:05.193980 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:05.193991 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:05.194426 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:05.194895 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:05.196610 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:05.198538 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:05.201316 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:05.202352 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:05.202539 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:05.207406 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:05.207413 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:05.207426 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:05.207465 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:05.210489 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:05.211564 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:05.211572 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:05.211579 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:05.211770 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:05.212229 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:05.212232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:05.212234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:05.212891 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:05.213024 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:05.213987 (+ 963us) server_negotiation.cc:299] Negotiation successful
0807 03:12:05.214400 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":350,"threads_started":1}
I20260807 03:12:05.215123 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:05.190346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:60595 (local address 127.17.31.3:34389)
0807 03:12:05.191022 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:05.191037 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:05.191078 (+ 41us) server_negotiation.cc:407] Connection header received
0807 03:12:05.193283 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:05.193294 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:05.193766 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:05.194115 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:05.196610 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:05.198538 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:05.201316 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:05.202362 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:05.202539 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:05.207654 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:05.207661 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:05.207671 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:05.207705 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:05.211454 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:05.212209 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:05.212218 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:05.212225 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:05.212397 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:05.212799 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:05.212802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:05.212803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:05.213421 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:05.213596 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:05.214224 (+ 628us) server_negotiation.cc:299] Negotiation successful
0807 03:12:05.214632 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":371,"threads_started":1}
I20260807 03:12:05.214871 1304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:05.190556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:37093 (local address 127.17.31.1:34945)
0807 03:12:05.191358 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:05.191376 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:05.191498 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:05.194888 (+ 3390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:05.194893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:05.195490 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:12:05.196400 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:05.196427 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:05.198750 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:05.198754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:05.201163 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:05.201179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:05.202572 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:05.202578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:05.203006 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:05.203884 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:12:05.203903 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:12:05.207131 (+ 3228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:12:05.210774 (+ 3643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:05.210784 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:05.210800 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:05.211405 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:05.211896 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:05.211899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:05.211901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:05.212095 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:05.213039 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:12:05.213047 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:12:05.213377 (+ 330us) client_negotiation.cc:769] Sending connection context
0807 03:12:05.213735 (+ 358us) client_negotiation.cc:240] Negotiation successful
0807 03:12:05.214233 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":356,"spinlock_wait_cycles":20224,"thread_start_us":590,"threads_started":1}
W20260807 03:12:05.218798 904 leader_election.cc:341] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389): Not found: Tablet not found: 0f68fb9f13b6488bac8595f545ba1619
W20260807 03:12:05.219115 904 leader_election.cc:341] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093): Not found: Tablet not found: 0f68fb9f13b6488bac8595f545ba1619
I20260807 03:12:05.219259 904 leader_election.cc:302] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbc8f9b4702f43ae934d6eae53ac2bae; no voters: 353ad0f0cbb2421d8478072cfd8a1059, 606f50ac7e8f496984dc801745d52cad
I20260807 03:12:05.219677 1302 raft_consensus.cc:2726] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:12:05.634709 1307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:05.633635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:52369)
0807 03:12:05.634182 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:05.634371 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":268,"threads_started":1}
I20260807 03:12:05.743044 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:05.742222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:60531)
0807 03:12:05.742756 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:05.742860 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":304,"threads_started":1}
I20260807 03:12:06.195955 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:06.195318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:43779)
0807 03:12:06.195731 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:06.195805 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":231,"threads_started":1}
I20260807 03:12:06.636301 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:06.635515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:47289)
0807 03:12:06.636045 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:06.636143 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":275,"threads_started":1}
I20260807 03:12:06.744616 1313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:06.743937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:55109)
0807 03:12:06.744363 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:06.744443 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":210,"threads_started":1}
I20260807 03:12:07.197511 1314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:07.196716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:38495)
0807 03:12:07.197250 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:07.197334 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":321,"threads_started":1}
I20260807 03:12:07.339896 1315 raft_consensus.cc:482] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:12:07.340085 1315 raft_consensus.cc:504] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } }
I20260807 03:12:07.340699 1315 leader_election.cc:288] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389), 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093)
W20260807 03:12:07.341696 904 leader_election.cc:341] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389): Not found: Tablet not found: 0f68fb9f13b6488bac8595f545ba1619
W20260807 03:12:07.341931 904 leader_election.cc:341] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093): Not found: Tablet not found: 0f68fb9f13b6488bac8595f545ba1619
I20260807 03:12:07.342037 904 leader_election.cc:302] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbc8f9b4702f43ae934d6eae53ac2bae; no voters: 353ad0f0cbb2421d8478072cfd8a1059, 606f50ac7e8f496984dc801745d52cad
I20260807 03:12:07.342336 1315 raft_consensus.cc:2726] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:12:07.637722 1316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:07.637067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:38447)
0807 03:12:07.637494 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:07.637568 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":228,"threads_started":1}
I20260807 03:12:07.746546 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:07.745511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:55429)
0807 03:12:07.746090 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:07.746226 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":240,"threads_started":1}
I20260807 03:12:08.199195 1318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:08.198453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:39603)
0807 03:12:08.198911 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:08.198992 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":243,"threads_started":1}
I20260807 03:12:08.639278 1319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:08.638580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:45391)
0807 03:12:08.639026 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:08.639103 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":227,"threads_started":1}
I20260807 03:12:08.748330 1320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:08.747500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:36379)
0807 03:12:08.748071 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:08.748151 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":296,"threads_started":1}
I20260807 03:12:09.200888 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:09.200221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:51935)
0807 03:12:09.200644 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:09.200724 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":230,"threads_started":1}
I20260807 03:12:09.506048 1322 raft_consensus.cc:482] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:12:09.506239 1322 raft_consensus.cc:504] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } }
I20260807 03:12:09.506924 1322 leader_election.cc:288] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389), 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093)
W20260807 03:12:09.507905 904 leader_election.cc:341] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389): Not found: Tablet not found: 0f68fb9f13b6488bac8595f545ba1619
W20260807 03:12:09.508162 904 leader_election.cc:341] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093): Not found: Tablet not found: 0f68fb9f13b6488bac8595f545ba1619
I20260807 03:12:09.508322 904 leader_election.cc:302] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbc8f9b4702f43ae934d6eae53ac2bae; no voters: 353ad0f0cbb2421d8478072cfd8a1059, 606f50ac7e8f496984dc801745d52cad
I20260807 03:12:09.508605 1322 raft_consensus.cc:2726] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:12:09.641085 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:09.640154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:56945)
0807 03:12:09.640670 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:09.640815 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":253,"threads_started":1}
I20260807 03:12:09.749971 1324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:09.749227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:38151)
0807 03:12:09.749700 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:09.749775 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":255,"threads_started":1}
I20260807 03:12:10.202644 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:10.201808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:48041)
0807 03:12:10.202252 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:10.202444 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":259,"threads_started":1}
I20260807 03:12:10.642582 1326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:10.641832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:36879)
0807 03:12:10.642277 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:10.642411 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":244,"threads_started":1}
I20260807 03:12:10.751502 1327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:10.750807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:55673)
0807 03:12:10.751240 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:10.751323 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":225,"threads_started":1}
I20260807 03:12:11.204378 1328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:11.203536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:39873)
0807 03:12:11.204086 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:11.204182 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":284,"threads_started":1}
I20260807 03:12:11.644243 1329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:11.643435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:38547)
0807 03:12:11.643957 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:11.644039 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":247,"threads_started":1}
I20260807 03:12:11.753037 1330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:11.752329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:35269)
0807 03:12:11.752794 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:11.752870 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":237,"threads_started":1}
I20260807 03:12:12.206154 1332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:12.205420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:43979)
0807 03:12:12.205875 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:12.205964 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":258,"threads_started":1}
I20260807 03:12:12.645961 1333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:12.645222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:39407)
0807 03:12:12.645697 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:12.645775 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":288,"threads_started":1}
I20260807 03:12:12.754623 1334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:12.753826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:52065)
0807 03:12:12.754283 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:12.754440 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":238,"threads_started":1}
I20260807 03:12:12.961032 1335 raft_consensus.cc:482] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:12:12.961233 1335 raft_consensus.cc:504] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } }
I20260807 03:12:12.961845 1335 leader_election.cc:288] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389), 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093)
W20260807 03:12:12.962826 904 leader_election.cc:341] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093): Not found: Tablet not found: 0f68fb9f13b6488bac8595f545ba1619
W20260807 03:12:12.963028 904 leader_election.cc:341] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389): Not found: Tablet not found: 0f68fb9f13b6488bac8595f545ba1619
I20260807 03:12:12.963105 904 leader_election.cc:302] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbc8f9b4702f43ae934d6eae53ac2bae; no voters: 353ad0f0cbb2421d8478072cfd8a1059, 606f50ac7e8f496984dc801745d52cad
I20260807 03:12:12.963325 1335 raft_consensus.cc:2726] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:12:13.207888 1336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:13.206998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:45665)
0807 03:12:13.207587 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:13.207675 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":337,"threads_started":1}
I20260807 03:12:13.647709 1337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:13.647019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:41795)
0807 03:12:13.647473 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:13.647560 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":215,"threads_started":1}
I20260807 03:12:13.756223 1338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:13.755560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:42585)
0807 03:12:13.755991 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:13.756067 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":225,"threads_started":1}
I20260807 03:12:14.209465 1339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:14.208784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:33211)
0807 03:12:14.209209 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:14.209288 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":223,"threads_started":1}
I20260807 03:12:14.649336 1340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:14.648621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:50093)
0807 03:12:14.649095 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:14.649168 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":291,"threads_started":1}
I20260807 03:12:14.757782 1341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:14.757051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:47371)
0807 03:12:14.757486 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:14.757562 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":218,"threads_started":1}
I20260807 03:12:15.211151 1342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:15.210461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:40305)
0807 03:12:15.210888 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:15.210965 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":232,"threads_started":1}
I20260807 03:12:15.651111 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:15.650421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:46497)
0807 03:12:15.650867 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:15.650948 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":226,"threads_started":1}
I20260807 03:12:15.759294 1344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:15.758569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:34423)
0807 03:12:15.758998 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:15.759097 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":242,"threads_started":1}
I20260807 03:12:16.212657 1345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:16.212017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:42565)
0807 03:12:16.212410 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:16.212486 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":204,"threads_started":1}
I20260807 03:12:16.652926 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:16.651965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:46181)
0807 03:12:16.652545 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:16.652647 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":292,"threads_started":1}
I20260807 03:12:16.760799 1347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:16.760100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:52409)
0807 03:12:16.760516 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:16.760634 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":224,"threads_started":1}
I20260807 03:12:17.214488 1348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:17.213680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:39501)
0807 03:12:17.214165 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:17.214252 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":239,"threads_started":1}
I20260807 03:12:17.387036 1349 raft_consensus.cc:482] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:12:17.387286 1349 raft_consensus.cc:504] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbc8f9b4702f43ae934d6eae53ac2bae" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 45557 } } peers { permanent_uuid: "353ad0f0cbb2421d8478072cfd8a1059" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 34389 } } peers { permanent_uuid: "606f50ac7e8f496984dc801745d52cad" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37093 } }
I20260807 03:12:17.388010 1349 leader_election.cc:288] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389), 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093)
W20260807 03:12:17.388950 904 leader_election.cc:341] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 353ad0f0cbb2421d8478072cfd8a1059 (127.17.31.3:34389): Not found: Tablet not found: 0f68fb9f13b6488bac8595f545ba1619
W20260807 03:12:17.389148 904 leader_election.cc:341] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 606f50ac7e8f496984dc801745d52cad (127.17.31.2:37093): Not found: Tablet not found: 0f68fb9f13b6488bac8595f545ba1619
I20260807 03:12:17.389241 904 leader_election.cc:302] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbc8f9b4702f43ae934d6eae53ac2bae; no voters: 353ad0f0cbb2421d8478072cfd8a1059, 606f50ac7e8f496984dc801745d52cad
I20260807 03:12:17.389467 1349 raft_consensus.cc:2726] T 0f68fb9f13b6488bac8595f545ba1619 P fbc8f9b4702f43ae934d6eae53ac2bae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:12:17.654645 1350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:17.653773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:58009)
0807 03:12:17.654259 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:17.654407 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":222,"threads_started":1}
I20260807 03:12:17.762687 1351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:17.761650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:37115)
0807 03:12:17.762352 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:17.762481 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":404,"threads_started":1}
I20260807 03:12:18.216041 1352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:18.215320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:59139)
0807 03:12:18.215768 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:18.215852 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":253,"threads_started":1}
I20260807 03:12:18.656191 1353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:18.655462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:48033)
0807 03:12:18.655913 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:18.655991 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":251,"threads_started":1}
I20260807 03:12:18.764616 1354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:18.763899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.1:56793)
0807 03:12:18.764344 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:18.764445 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":236,"threads_started":1}
I20260807 03:12:19.217909 1355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:19.217157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.2:48569)
0807 03:12:19.217663 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:19.217746 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":287,"threads_started":1}
I20260807 03:12:19.635310 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 848
I20260807 03:12:19.650959 1356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:19.636345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38110 (local address 127.17.31.1:45557)
0807 03:12:19.636989 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:19.637017 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:19.637049 (+ 32us) server_negotiation.cc:407] Connection header received
0807 03:12:19.637131 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:19.637161 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:19.637490 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:19.637680 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:19.639834 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:19.640927 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:19.642169 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:19.642827 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:19.642927 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:19.645557 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:19.645568 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:19.645581 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:19.645619 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:19.648511 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:19.649066 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:19.649073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:19.649075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:19.649207 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:19.649507 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:19.649510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:19.649512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:19.649863 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:19.650085 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:19.650388 (+ 303us) server_negotiation.cc:299] Negotiation successful
0807 03:12:19.650585 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":267,"threads_started":1}
I20260807 03:12:19.652278 1016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:12:19.658001 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:19.657254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35575 (local address 127.17.31.3:37773)
0807 03:12:19.657734 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:19.657856 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:35575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":265,"threads_started":1}
I20260807 03:12:19.742831 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 848
I20260807 03:12:19.763150 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 1025
I20260807 03:12:19.776983 1360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:19.763878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33912 (local address 127.17.31.2:37093)
0807 03:12:19.764417 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:19.764422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:19.764437 (+ 15us) server_negotiation.cc:407] Connection header received
0807 03:12:19.764491 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:19.764494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:19.764561 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:19.764692 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:19.766187 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:19.767051 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:19.768314 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:19.768730 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:19.768866 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:19.771660 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:19.771665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:19.771669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:19.771702 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:19.774229 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:19.774860 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:19.774866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:19.774871 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:19.775010 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:19.775412 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:19.775419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:19.775424 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:19.775706 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:19.776017 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:19.776355 (+ 338us) server_negotiation.cc:299] Negotiation successful
0807 03:12:19.776617 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":264,"threads_started":1}
I20260807 03:12:19.777808 1155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:12:19.860180 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 1025
I20260807 03:12:19.878746 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 1163
I20260807 03:12:19.891789 1362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:19.879505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.17.31.3:34389)
0807 03:12:19.880053 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:19.880058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:19.880076 (+ 18us) server_negotiation.cc:407] Connection header received
0807 03:12:19.880129 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:19.880133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:19.880202 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:19.880295 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:19.881689 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:19.882539 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:19.883727 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:19.884130 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:19.884222 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:19.886765 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:19.886770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:19.886773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:19.886802 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:19.889421 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:19.890121 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:19.890124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:19.890126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:19.890269 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:19.890655 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:19.890658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:19.890660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:19.890896 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:19.891091 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:19.891344 (+ 253us) server_negotiation.cc:299] Negotiation successful
0807 03:12:19.891497 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":287,"threads_started":1}
I20260807 03:12:19.892540 1291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:12:19.975989 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 1163
2026-08-07T03:12:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24388 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:12:20 dist-test-slave-tm1g krb5kdc[1366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:12:20 dist-test-slave-tm1g krb5kdc[1366](info): set up 2 sockets
Aug 07 03:12:20 dist-test-slave-tm1g krb5kdc[1366](info): commencing operation
krb5kdc: starting...
W20260807 03:12:21.088356 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.012s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:12:21 dist-test-slave-tm1g krb5kdc[1366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:12:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:12:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:12:21.317622 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:40745
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:40745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:12:21.639225 1379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:12:21.639565 1379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:12:21.639720 1379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:12:21.646767 1379 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 03:12:21.646888 1379 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 03:12:21.646976 1379 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 03:12:21.647074 1379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:12:21.647166 1379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:12:21.647264 1379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:12:21.647356 1379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:12:21.647434 1379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:12:21.656646 1379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:40745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:40745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:12:21.657835 1379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:12:21.659135 1379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:12:21.670599 1387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:21.670683 1384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:21.670845 1385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:12:21.671957 1379 server_base.cc:1034] running on GCE node
I20260807 03:12:21.672776 1379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:12:21.674130 1379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:12:21.675385 1379 hybrid_clock.cc:648] HybridClock initialized: now 1786072341675313 us; error 48 us; skew 500 ppm
I20260807 03:12:21.675719 1379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:12:21 dist-test-slave-tm1g krb5kdc[1366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072341, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:12:21.678776 1379 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:12:21.680173 1379 webserver.cc:460] Webserver started at http://127.17.31.62:39643/ using document root <none> and password file <none>
I20260807 03:12:21.680804 1379 fs_manager.cc:359] Metadata directory not provided
I20260807 03:12:21.680888 1379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:12:21.681142 1379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:12:21.683959 1379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5"
format_stamp: "Formatted at 2026-08-07 03:12:21 on dist-test-slave-tm1g"
server_key: "98e20d169a6ae0a6cda855650e29bb02"
server_key_iv: "7cfab152effebceb1788f88e4797c43f"
server_key_version: "encryptionkey@0"
I20260807 03:12:21.684716 1379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5"
format_stamp: "Formatted at 2026-08-07 03:12:21 on dist-test-slave-tm1g"
server_key: "98e20d169a6ae0a6cda855650e29bb02"
server_key_iv: "7cfab152effebceb1788f88e4797c43f"
server_key_version: "encryptionkey@0"
I20260807 03:12:21.689613 1379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:12:21.692925 1394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:21.694231 1379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:12:21.694445 1379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5"
format_stamp: "Formatted at 2026-08-07 03:12:21 on dist-test-slave-tm1g"
server_key: "98e20d169a6ae0a6cda855650e29bb02"
server_key_iv: "7cfab152effebceb1788f88e4797c43f"
server_key_version: "encryptionkey@0"
I20260807 03:12:21.694635 1379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:12:21.713951 1379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:12:21.717937 1379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:12:21.718158 1379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:12:21.736557 1379 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:40745
I20260807 03:12:21.736588 1450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:40745 every 8 connection(s)
I20260807 03:12:21.737892 1379 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:12:21.739338 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 1379
I20260807 03:12:21.739588 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:12:21.740537 17532 external_mini_cluster.cc:1428] Setting key b2c8273cb040ca8ce7827f4f24039128
I20260807 03:12:21.742017 1451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 03:12:21 dist-test-slave-tm1g krb5kdc[1366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:12:21.758167 1451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Bootstrap starting.
I20260807 03:12:21.762413 1451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Neither blocks nor log segments found. Creating new log.
I20260807 03:12:21.763738 1451 log.cc:824] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Log is configured to *not* fsync() on all Append() calls
I20260807 03:12:21.766884 1451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: No bootstrap required, opened a new log
I20260807 03:12:21.767668 1454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:21.743035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.17.31.62:40745)
0807 03:12:21.744002 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:21.744038 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:21.744094 (+ 56us) server_negotiation.cc:407] Connection header received
0807 03:12:21.746377 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:21.746404 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:21.746915 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:21.747425 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:21.749773 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:21.751239 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:21.753149 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:21.753861 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:21.754007 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:21.758944 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:21.758955 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:21.758972 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:21.759015 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:21.763078 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:21.763846 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:21.763854 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:21.763864 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:21.764069 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:21.764488 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:21.764494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:21.764498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:21.765116 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:21.765491 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:21.766007 (+ 516us) server_negotiation.cc:299] Negotiation successful
0807 03:12:21.766494 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":402,"threads_started":1}
I20260807 03:12:21.775151 1451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } }
I20260807 03:12:21.775497 1451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:12:21.775614 1451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a53acbae1f24a0e97f0a0e4a7b884f5, State: Initialized, Role: FOLLOWER
I20260807 03:12:21.776209 1451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } }
I20260807 03:12:21.776489 1451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:12:21.776666 1451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:12:21.776876 1451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:12:21.779177 1451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } }
I20260807 03:12:21.779706 1451 leader_election.cc:302] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a53acbae1f24a0e97f0a0e4a7b884f5; no voters:
I20260807 03:12:21.780311 1451 leader_election.cc:288] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:12:21.780575 1456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:12:21.782282 1456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4a53acbae1f24a0e97f0a0e4a7b884f5, State: Running, Role: LEADER
I20260807 03:12:21.782817 1451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:12:21.782922 1456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } }
I20260807 03:12:21.786204 1457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } } }
I20260807 03:12:21.786226 1458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a53acbae1f24a0e97f0a0e4a7b884f5. Latest consensus state: current_term: 1 leader_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } } }
I20260807 03:12:21.786866 1457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [sys.catalog]: This master's current role is: LEADER
I20260807 03:12:21.786878 1458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [sys.catalog]: This master's current role is: LEADER
I20260807 03:12:21.788461 1466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:12:21.793467 1466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:12:21.802085 1466 catalog_manager.cc:1347] Generated new cluster ID: b7b3db2bd55248b39d55a906a1aadc35
I20260807 03:12:21.802201 1466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:12:21.827996 1466 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:12:21.830025 1466 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:12:21.841717 1466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Generated new TSK 0
I20260807 03:12:21.842762 1466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:12:21.912357 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:40745
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:12:22.244244 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:12:22.244539 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:12:22.244697 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:12:22.251756 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:12:22.251895 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:12:22.252074 1478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:12:22.261286 1478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:40745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:12:22.262483 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:12:22.263757 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:12:22.276635 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:22.276702 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:22.276731 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:12:22.277516 1478 server_base.cc:1034] running on GCE node
I20260807 03:12:22.278034 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:12:22.278978 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:12:22.280170 1478 hybrid_clock.cc:648] HybridClock initialized: now 1786072342280120 us; error 50 us; skew 500 ppm
I20260807 03:12:22.280437 1478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:12:22 dist-test-slave-tm1g krb5kdc[1366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072342, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:12:22.286808 1478 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:12:22.288246 1478 webserver.cc:460] Webserver started at http://127.17.31.1:36277/ using document root <none> and password file <none>
I20260807 03:12:22.288946 1478 fs_manager.cc:359] Metadata directory not provided
I20260807 03:12:22.289085 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:12:22.289418 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:12:22.292281 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "d6a772b69d794e7b87f94cc0af808b20"
format_stamp: "Formatted at 2026-08-07 03:12:22 on dist-test-slave-tm1g"
server_key: "bea1f651ca29b029ac52866a3bccfe5f"
server_key_iv: "af02a5e05b7a2700e11327014b2b0573"
server_key_version: "encryptionkey@0"
I20260807 03:12:22.293009 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "d6a772b69d794e7b87f94cc0af808b20"
format_stamp: "Formatted at 2026-08-07 03:12:22 on dist-test-slave-tm1g"
server_key: "bea1f651ca29b029ac52866a3bccfe5f"
server_key_iv: "af02a5e05b7a2700e11327014b2b0573"
server_key_version: "encryptionkey@0"
I20260807 03:12:22.297834 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260807 03:12:22.301190 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:22.302547 1478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:12:22.302667 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "d6a772b69d794e7b87f94cc0af808b20"
format_stamp: "Formatted at 2026-08-07 03:12:22 on dist-test-slave-tm1g"
server_key: "bea1f651ca29b029ac52866a3bccfe5f"
server_key_iv: "af02a5e05b7a2700e11327014b2b0573"
server_key_version: "encryptionkey@0"
I20260807 03:12:22.302812 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:12:22.342703 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:12:22.346989 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:12:22.347229 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:12:22.348133 1478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:12:22.349390 1478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:12:22.349473 1478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:22.349555 1478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:12:22.349598 1478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:22.382965 1478 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:42549
I20260807 03:12:22.383015 1613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:42549 every 8 connection(s)
I20260807 03:12:22.384289 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:12:22.390223 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 1478
I20260807 03:12:22.390491 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:12:22.391148 17532 external_mini_cluster.cc:1428] Setting key 948bdc7be0039a038678ac4011e6d475
Aug 07 03:12:22 dist-test-slave-tm1g krb5kdc[1366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072342, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:12:22.405627 1616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:22.387042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:43027 (local address 127.17.31.62:40745)
0807 03:12:22.387542 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:22.387547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:22.388625 (+ 1078us) server_negotiation.cc:407] Connection header received
0807 03:12:22.390567 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:22.390573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:22.390667 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:22.390791 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:22.392449 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:22.393325 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:22.394942 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:22.395334 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:22.395470 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:22.398858 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:22.398867 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:22.398874 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:22.398919 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:22.402452 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:22.403250 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:22.403254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:22.403256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:22.403378 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:22.403856 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:22.403859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:22.403861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:22.404111 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:22.404229 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:22.405072 (+ 843us) server_negotiation.cc:299] Negotiation successful
0807 03:12:22.405222 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":270,"threads_started":1}
I20260807 03:12:22.407068 1617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:22.387739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:43027)
0807 03:12:22.388449 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:22.388485 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:22.390225 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:22.391028 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:22.391035 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:22.391590 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:12:22.392278 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:22.392296 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:22.393483 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:22.393487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:22.394799 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:22.394815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:22.395559 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:22.395568 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:22.395786 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:22.396458 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:12:22.396479 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:12:22.398641 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:12:22.402595 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:22.402601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:22.402611 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:22.403120 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:22.403484 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:22.403490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:22.403495 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:22.403736 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:22.404246 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:12:22.404256 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:12:22.404673 (+ 417us) client_negotiation.cc:769] Sending connection context
0807 03:12:22.404979 (+ 306us) client_negotiation.cc:240] Negotiation successful
0807 03:12:22.405559 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":305,"threads_started":1}
I20260807 03:12:22.408522 1614 heartbeater.cc:347] Connected to a master server at 127.17.31.62:40745
I20260807 03:12:22.408808 1614 heartbeater.cc:464] Registering TS with master...
I20260807 03:12:22.409466 1614 heartbeater.cc:511] Master 127.17.31.62:40745 requested a full tablet report, sending...
I20260807 03:12:22.411512 1412 ts_manager.cc:194] Registered new tserver with Master: d6a772b69d794e7b87f94cc0af808b20 (127.17.31.1:42549)
I20260807 03:12:22.412920 1412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:43027
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:12:22.450063 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:40745
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:12:22.791941 1620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:12:22.792203 1620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:12:22.792299 1620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:12:22.799652 1620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:12:22.799749 1620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:12:22.799860 1620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:12:22.809257 1620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:40745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:12:22.810642 1620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:12:22.811874 1620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:12:22.825796 1628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:22.825822 1626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:22.825904 1625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:12:22.831712 1620 server_base.cc:1034] running on GCE node
I20260807 03:12:22.832451 1620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:12:22.833501 1620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:12:22.834746 1620 hybrid_clock.cc:648] HybridClock initialized: now 1786072342834710 us; error 48 us; skew 500 ppm
I20260807 03:12:22.835083 1620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:12:22 dist-test-slave-tm1g krb5kdc[1366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072342, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:12:22.838455 1620 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:12:22.840088 1620 webserver.cc:460] Webserver started at http://127.17.31.2:41107/ using document root <none> and password file <none>
I20260807 03:12:22.840881 1620 fs_manager.cc:359] Metadata directory not provided
I20260807 03:12:22.841027 1620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:12:22.841339 1620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:12:22.844246 1620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "059b6926da8c42f6bc96c7e97a141363"
format_stamp: "Formatted at 2026-08-07 03:12:22 on dist-test-slave-tm1g"
server_key: "6e36f5fc9379f5b40cf60550ff93416a"
server_key_iv: "9d2804ca423099098de6e6cd9803c751"
server_key_version: "encryptionkey@0"
I20260807 03:12:22.844986 1620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "059b6926da8c42f6bc96c7e97a141363"
format_stamp: "Formatted at 2026-08-07 03:12:22 on dist-test-slave-tm1g"
server_key: "6e36f5fc9379f5b40cf60550ff93416a"
server_key_iv: "9d2804ca423099098de6e6cd9803c751"
server_key_version: "encryptionkey@0"
I20260807 03:12:22.849958 1620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:12:22.853395 1635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:22.854782 1620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:12:22.854969 1620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "059b6926da8c42f6bc96c7e97a141363"
format_stamp: "Formatted at 2026-08-07 03:12:22 on dist-test-slave-tm1g"
server_key: "6e36f5fc9379f5b40cf60550ff93416a"
server_key_iv: "9d2804ca423099098de6e6cd9803c751"
server_key_version: "encryptionkey@0"
I20260807 03:12:22.855180 1620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:12:22.896239 1620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:12:22.900632 1620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:12:22.900935 1620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:12:22.901885 1620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:12:22.903225 1620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:12:22.903355 1620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:22.903517 1620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:12:22.903569 1620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:22.936461 1620 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:34983
I20260807 03:12:22.936502 1753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:34983 every 8 connection(s)
I20260807 03:12:22.937714 1620 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:12:22.944188 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 1620
I20260807 03:12:22.944375 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:12:22.944864 17532 external_mini_cluster.cc:1428] Setting key 441cdfd6b953df9e26dc2f7ad5b96b40
Aug 07 03:12:22 dist-test-slave-tm1g krb5kdc[1366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072342, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:12:22.959260 1757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:22.940357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:48869 (local address 127.17.31.62:40745)
0807 03:12:22.940862 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:22.940869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:22.941731 (+ 862us) server_negotiation.cc:407] Connection header received
0807 03:12:22.943538 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:22.943543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:22.943610 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:22.943706 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:22.945333 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:22.946292 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:22.948116 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:22.948909 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:22.949056 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:22.952389 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:22.952399 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:22.952407 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:22.952453 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:22.955993 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:22.956679 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:22.956687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:22.956692 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:22.956852 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:22.957238 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:22.957245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:22.957252 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:22.957564 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:22.957676 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:22.958646 (+ 970us) server_negotiation.cc:299] Negotiation successful
0807 03:12:22.958825 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":367,"thread_start_us":241,"threads_started":1}
I20260807 03:12:22.960597 1758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:22.940852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:48869)
0807 03:12:22.941563 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:22.941606 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:22.943284 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:22.943916 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:22.943923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:22.944538 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:12:22.945137 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:22.945155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:22.946526 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:22.946531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:22.947963 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:22.947980 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:22.949108 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:22.949112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:22.949302 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:22.949972 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:12:22.949994 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:12:22.952155 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:12:22.956129 (+ 3974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:22.956135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:22.956146 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:22.956562 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:22.956959 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:22.956962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:22.956965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:22.957133 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:22.957704 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:12:22.957713 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:12:22.958130 (+ 417us) client_negotiation.cc:769] Sending connection context
0807 03:12:22.958503 (+ 373us) client_negotiation.cc:240] Negotiation successful
0807 03:12:22.959055 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":276,"threads_started":1}
I20260807 03:12:22.962006 1754 heartbeater.cc:347] Connected to a master server at 127.17.31.62:40745
I20260807 03:12:22.962378 1754 heartbeater.cc:464] Registering TS with master...
I20260807 03:12:22.963088 1754 heartbeater.cc:511] Master 127.17.31.62:40745 requested a full tablet report, sending...
I20260807 03:12:22.964568 1412 ts_manager.cc:194] Registered new tserver with Master: 059b6926da8c42f6bc96c7e97a141363 (127.17.31.2:34983)
I20260807 03:12:22.965389 1412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:48869
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:12:23.003648 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:40745
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:12:23.329718 1761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:12:23.329998 1761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:12:23.330160 1761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:12:23.337038 1761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:12:23.337147 1761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:12:23.337371 1761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:12:23.346639 1761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:40745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:12:23.347913 1761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:12:23.349186 1761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:12:23.361730 1769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:23.361799 1767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:23.361783 1766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:12:23.362462 1761 server_base.cc:1034] running on GCE node
I20260807 03:12:23.363041 1761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:12:23.363874 1761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:12:23.365123 1761 hybrid_clock.cc:648] HybridClock initialized: now 1786072343365081 us; error 50 us; skew 500 ppm
I20260807 03:12:23.365435 1761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:12:23 dist-test-slave-tm1g krb5kdc[1366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072343, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:12:23.368438 1761 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:12:23.369777 1761 webserver.cc:460] Webserver started at http://127.17.31.3:38191/ using document root <none> and password file <none>
I20260807 03:12:23.370502 1761 fs_manager.cc:359] Metadata directory not provided
I20260807 03:12:23.370649 1761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:12:23.370952 1761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:12:23.373714 1761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "65a813aa28bc4313883d12f358cdd313"
format_stamp: "Formatted at 2026-08-07 03:12:23 on dist-test-slave-tm1g"
server_key: "1d4f9c6a3bd0742703b2014955c7d015"
server_key_iv: "98534258f4ee3cfdedbf00333722a11c"
server_key_version: "encryptionkey@0"
I20260807 03:12:23.374536 1761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "65a813aa28bc4313883d12f358cdd313"
format_stamp: "Formatted at 2026-08-07 03:12:23 on dist-test-slave-tm1g"
server_key: "1d4f9c6a3bd0742703b2014955c7d015"
server_key_iv: "98534258f4ee3cfdedbf00333722a11c"
server_key_version: "encryptionkey@0"
I20260807 03:12:23.379310 1761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 03:12:23.382617 1776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:23.384207 1761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:12:23.384356 1761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "65a813aa28bc4313883d12f358cdd313"
format_stamp: "Formatted at 2026-08-07 03:12:23 on dist-test-slave-tm1g"
server_key: "1d4f9c6a3bd0742703b2014955c7d015"
server_key_iv: "98534258f4ee3cfdedbf00333722a11c"
server_key_version: "encryptionkey@0"
I20260807 03:12:23.384596 1761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:12:23.415719 1614 heartbeater.cc:503] Master 127.17.31.62:40745 was elected leader, sending a full tablet report...
I20260807 03:12:23.417452 1761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:12:23.421043 1761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:12:23.421293 1761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:12:23.422143 1761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:12:23.423375 1761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:12:23.423446 1761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:23.423525 1761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:12:23.423575 1761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:23.456300 1761 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:36549
I20260807 03:12:23.456354 1891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:36549 every 8 connection(s)
I20260807 03:12:23.457504 1761 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:12:23.465723 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 1761
I20260807 03:12:23.465934 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:12:23.466513 17532 external_mini_cluster.cc:1428] Setting key 3765b64011fa5e0d29982b637fedfa3f
Aug 07 03:12:23 dist-test-slave-tm1g krb5kdc[1366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072343, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:12:23.477955 1894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:23.460073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:43763 (local address 127.17.31.62:40745)
0807 03:12:23.460553 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:23.460558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:23.461378 (+ 820us) server_negotiation.cc:407] Connection header received
0807 03:12:23.463292 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:23.463297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:23.463391 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:23.463514 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:23.464915 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.466031 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.467991 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.468616 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.468744 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:23.472249 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:23.472262 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:23.472266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:23.472295 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:23.475051 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:23.475742 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:23.475751 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:23.475753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:23.475875 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:23.476321 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:23.476325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:23.476330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:23.476587 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:23.476757 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:23.477387 (+ 630us) server_negotiation.cc:299] Negotiation successful
0807 03:12:23.477598 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":263,"threads_started":1}
I20260807 03:12:23.478886 1895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:23.460512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:43763)
0807 03:12:23.461206 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:23.461248 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:23.463001 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:23.463744 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:23.463752 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:23.464243 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:12:23.464746 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:23.464761 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.466196 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.466200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:23.467806 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:23.467822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.468856 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.468864 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:23.469155 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:23.469918 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:12:23.469939 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:12:23.472051 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:12:23.475193 (+ 3142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:23.475201 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:23.475215 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:23.475609 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:23.475995 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:23.476000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:23.476004 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:23.476188 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:23.476720 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:12:23.476727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:12:23.477032 (+ 305us) client_negotiation.cc:769] Sending connection context
0807 03:12:23.477273 (+ 241us) client_negotiation.cc:240] Negotiation successful
0807 03:12:23.477722 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":298,"threads_started":1}
I20260807 03:12:23.480283 1892 heartbeater.cc:347] Connected to a master server at 127.17.31.62:40745
I20260807 03:12:23.480631 1892 heartbeater.cc:464] Registering TS with master...
I20260807 03:12:23.481484 1892 heartbeater.cc:511] Master 127.17.31.62:40745 requested a full tablet report, sending...
I20260807 03:12:23.483027 1412 ts_manager.cc:194] Registered new tserver with Master: 65a813aa28bc4313883d12f358cdd313 (127.17.31.3:36549)
I20260807 03:12:23.483850 1412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:43763
I20260807 03:12:23.493831 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:12:23.511308 1894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:23.498402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.17.31.62:40745)
0807 03:12:23.498588 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:23.498597 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:23.499121 (+ 524us) server_negotiation.cc:407] Connection header received
0807 03:12:23.499246 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:23.499249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:23.499319 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:23.499396 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:23.500877 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.501621 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.503056 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.503452 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.503546 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:23.505876 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:23.505881 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:23.505884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:23.505912 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:23.508871 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:23.509564 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:23.509568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:23.509570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:23.509680 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:23.510028 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:23.510032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:23.510034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:23.510404 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:23.510592 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:23.510852 (+ 260us) server_negotiation.cc:299] Negotiation successful
0807 03:12:23.511004 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 03:12:23.515139 1412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:12:23.522015 1412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:12:23.544171 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:23.534612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.17.31.3:36549)
0807 03:12:23.535441 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:23.535451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:23.535488 (+ 37us) server_negotiation.cc:407] Connection header received
0807 03:12:23.535609 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:23.535618 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:23.535856 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:23.536038 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:12:23.537445 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.539118 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.541531 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.542624 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.542757 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:23.542995 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:23.543648 (+ 653us) server_negotiation.cc:299] Negotiation successful
0807 03:12:23.543810 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":497,"threads_started":1}
I20260807 03:12:23.544193 1905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:23.533831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:36549 (local address 127.0.0.1:42494)
0807 03:12:23.535213 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:23.535230 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:23.535338 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:23.536374 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:23.536383 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:23.536424 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:12:23.537161 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:23.537175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.539293 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.539303 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:23.541373 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:23.541386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.543164 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.543171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:23.543368 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:23.543416 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 03:12:23.543556 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 03:12:23.543738 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1259,"mutex_wait_us":246,"spinlock_wait_cycles":896,"thread_start_us":338,"threads_started":1}
I20260807 03:12:23.545087 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:23.533862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38132 (local address 127.17.31.2:34983)
0807 03:12:23.535798 (+ 1936us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:23.535806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:23.535832 (+ 26us) server_negotiation.cc:407] Connection header received
0807 03:12:23.535922 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:23.535929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:23.536159 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:23.536501 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:12:23.537444 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.539129 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.542293 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.543787 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.543909 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:23.544086 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:23.544760 (+ 674us) server_negotiation.cc:299] Negotiation successful
0807 03:12:23.544879 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1782,"thread_start_us":1678,"threads_started":1}
I20260807 03:12:23.545902 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:23.533650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:34983 (local address 127.0.0.1:38132)
0807 03:12:23.534561 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:23.534602 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:23.534780 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:23.536375 (+ 1595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:23.536383 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:23.536432 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:12:23.537165 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:23.537175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.539293 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.539303 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:23.542127 (+ 2824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:23.542142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.544244 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.544251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:23.544461 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:23.544551 (+ 90us) client_negotiation.cc:769] Sending connection context
0807 03:12:23.544641 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 03:12:23.545474 (+ 833us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":1114,"spinlock_wait_cycles":6784,"thread_start_us":1234,"threads_started":1}
I20260807 03:12:23.546180 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:23.535867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.17.31.1:42549)
0807 03:12:23.536805 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:23.536813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:23.536839 (+ 26us) server_negotiation.cc:407] Connection header received
0807 03:12:23.536931 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:23.536938 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:23.537114 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:23.537323 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:12:23.537904 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.539070 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.542443 (+ 3373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.543933 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.544057 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:23.544270 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:23.545645 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0807 03:12:23.545806 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":850,"threads_started":1}
I20260807 03:12:23.546198 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:23.535271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:42549 (local address 127.0.0.1:37416)
0807 03:12:23.536076 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:23.536095 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:23.536204 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:23.537320 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:23.537325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:23.537354 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:12:23.537733 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:23.537743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.539245 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.539250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:23.542140 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:23.542150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.544446 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.544452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:23.545434 (+ 982us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:23.545477 (+ 43us) client_negotiation.cc:769] Sending connection context
0807 03:12:23.545582 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 03:12:23.545719 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":238,"thread_start_us":799,"threads_started":1}
I20260807 03:12:23.548486 1824 tablet_service.cc:1467] Processing CreateTablet for tablet ea41e668f055400ca3b64b4201bd128c (DEFAULT_TABLE table=test-table [id=dc5992ae492745a78f1d20e7a5fc2c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:12:23.549329 1685 tablet_service.cc:1467] Processing CreateTablet for tablet ea41e668f055400ca3b64b4201bd128c (DEFAULT_TABLE table=test-table [id=dc5992ae492745a78f1d20e7a5fc2c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:12:23.550077 1824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea41e668f055400ca3b64b4201bd128c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:12:23.549768 1543 tablet_service.cc:1467] Processing CreateTablet for tablet ea41e668f055400ca3b64b4201bd128c (DEFAULT_TABLE table=test-table [id=dc5992ae492745a78f1d20e7a5fc2c9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:12:23.550525 1685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea41e668f055400ca3b64b4201bd128c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:12:23.551120 1543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea41e668f055400ca3b64b4201bd128c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:12:23.562101 1911 tablet_bootstrap.cc:492] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363: Bootstrap starting.
I20260807 03:12:23.564626 1912 tablet_bootstrap.cc:492] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313: Bootstrap starting.
I20260807 03:12:23.566915 1911 tablet_bootstrap.cc:654] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363: Neither blocks nor log segments found. Creating new log.
I20260807 03:12:23.566980 1913 tablet_bootstrap.cc:492] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20: Bootstrap starting.
I20260807 03:12:23.568116 1912 tablet_bootstrap.cc:654] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313: Neither blocks nor log segments found. Creating new log.
I20260807 03:12:23.568377 1911 log.cc:824] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363: Log is configured to *not* fsync() on all Append() calls
I20260807 03:12:23.569342 1912 log.cc:824] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313: Log is configured to *not* fsync() on all Append() calls
I20260807 03:12:23.570145 1913 tablet_bootstrap.cc:654] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20: Neither blocks nor log segments found. Creating new log.
I20260807 03:12:23.571116 1913 log.cc:824] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20: Log is configured to *not* fsync() on all Append() calls
I20260807 03:12:23.572036 1911 tablet_bootstrap.cc:492] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363: No bootstrap required, opened a new log
I20260807 03:12:23.572402 1911 ts_tablet_manager.cc:1397] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260807 03:12:23.573613 1913 tablet_bootstrap.cc:492] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20: No bootstrap required, opened a new log
I20260807 03:12:23.574136 1913 ts_tablet_manager.cc:1397] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260807 03:12:23.576838 1912 tablet_bootstrap.cc:492] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313: No bootstrap required, opened a new log
I20260807 03:12:23.577105 1912 ts_tablet_manager.cc:1397] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260807 03:12:23.581578 1913 raft_consensus.cc:348] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } } peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } }
I20260807 03:12:23.581928 1913 raft_consensus.cc:374] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:12:23.582051 1913 raft_consensus.cc:729] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a772b69d794e7b87f94cc0af808b20, State: Initialized, Role: FOLLOWER
I20260807 03:12:23.582687 1913 consensus_queue.cc:260] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } } peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } }
I20260807 03:12:23.584327 1913 ts_tablet_manager.cc:1428] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20: Time spent starting tablet: real 0.010s user 0.003s sys 0.006s
I20260807 03:12:23.583729 1911 raft_consensus.cc:348] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } } peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } }
I20260807 03:12:23.585187 1911 raft_consensus.cc:374] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:12:23.585351 1911 raft_consensus.cc:729] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059b6926da8c42f6bc96c7e97a141363, State: Initialized, Role: FOLLOWER
I20260807 03:12:23.585961 1911 consensus_queue.cc:260] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } } peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } }
I20260807 03:12:23.587170 1754 heartbeater.cc:503] Master 127.17.31.62:40745 was elected leader, sending a full tablet report...
I20260807 03:12:23.587767 1911 ts_tablet_manager.cc:1428] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260807 03:12:23.588583 1912 raft_consensus.cc:348] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } } peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } }
I20260807 03:12:23.588977 1912 raft_consensus.cc:374] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:12:23.589079 1912 raft_consensus.cc:729] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a813aa28bc4313883d12f358cdd313, State: Initialized, Role: FOLLOWER
I20260807 03:12:23.589668 1912 consensus_queue.cc:260] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } } peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } }
I20260807 03:12:23.590791 1892 heartbeater.cc:503] Master 127.17.31.62:40745 was elected leader, sending a full tablet report...
I20260807 03:12:23.592376 1912 ts_tablet_manager.cc:1428] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260807 03:12:23.619207 1918 raft_consensus.cc:482] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:12:23.619508 1918 raft_consensus.cc:504] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } } peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } }
I20260807 03:12:23.621094 1918 leader_election.cc:288] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65a813aa28bc4313883d12f358cdd313 (127.17.31.3:36549), d6a772b69d794e7b87f94cc0af808b20 (127.17.31.1:42549)
I20260807 03:12:23.628080 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:23.621337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:57101 (local address 127.17.31.3:36549)
0807 03:12:23.621525 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:23.621534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:23.622592 (+ 1058us) server_negotiation.cc:407] Connection header received
0807 03:12:23.622745 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:23.622751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:23.622827 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:23.622911 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:12:23.623537 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.624328 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.626241 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.627094 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.627192 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:23.627263 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:23.627696 (+ 433us) server_negotiation.cc:299] Negotiation successful
0807 03:12:23.627840 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 03:12:23.628171 1920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:23.621474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:36549 (local address 127.17.31.2:57101)
0807 03:12:23.622480 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:23.622496 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:23.622610 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:23.622940 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:23.622944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:23.622967 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:12:23.623387 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:23.623396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.624468 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.624475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:23.626081 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:23.626093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.627322 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.627328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:23.627499 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:23.627544 (+ 45us) client_negotiation.cc:769] Sending connection context
0807 03:12:23.627644 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 03:12:23.627733 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":35,"thread_start_us":532,"threads_started":1}
I20260807 03:12:23.628252 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:23.621623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:57421 (local address 127.17.31.1:42549)
0807 03:12:23.621871 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:23.621880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:23.622723 (+ 843us) server_negotiation.cc:407] Connection header received
0807 03:12:23.622864 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:23.622868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:23.622957 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:23.623052 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:12:23.623651 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.624442 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.626438 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.627299 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.627435 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:23.627535 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:23.627861 (+ 326us) server_negotiation.cc:299] Negotiation successful
0807 03:12:23.627993 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 03:12:23.628604 1921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:23.621543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:42549 (local address 127.17.31.2:57421)
0807 03:12:23.622590 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:23.622608 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:23.622745 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:23.623086 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:23.623089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:23.623118 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:12:23.623494 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:23.623506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.624589 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.624592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:23.626105 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:23.626118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.627505 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.627510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:23.627678 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:23.627711 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 03:12:23.627802 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 03:12:23.627934 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":20,"thread_start_us":548,"threads_started":1}
I20260807 03:12:23.629406 1844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea41e668f055400ca3b64b4201bd128c" candidate_uuid: "059b6926da8c42f6bc96c7e97a141363" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a813aa28bc4313883d12f358cdd313" is_pre_election: true
I20260807 03:12:23.629592 1564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea41e668f055400ca3b64b4201bd128c" candidate_uuid: "059b6926da8c42f6bc96c7e97a141363" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a772b69d794e7b87f94cc0af808b20" is_pre_election: true
I20260807 03:12:23.629885 1844 raft_consensus.cc:2454] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059b6926da8c42f6bc96c7e97a141363 in term 0.
I20260807 03:12:23.630136 1564 raft_consensus.cc:2454] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059b6926da8c42f6bc96c7e97a141363 in term 0.
I20260807 03:12:23.630611 1636 leader_election.cc:302] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059b6926da8c42f6bc96c7e97a141363, 65a813aa28bc4313883d12f358cdd313; no voters:
I20260807 03:12:23.631156 1918 raft_consensus.cc:2781] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:12:23.631294 1918 raft_consensus.cc:482] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:12:23.631413 1918 raft_consensus.cc:3037] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:12:23.633965 1918 raft_consensus.cc:504] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } } peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } }
I20260807 03:12:23.634729 1918 leader_election.cc:288] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [CANDIDATE]: Term 1 election: Requested vote from peers 65a813aa28bc4313883d12f358cdd313 (127.17.31.3:36549), d6a772b69d794e7b87f94cc0af808b20 (127.17.31.1:42549)
I20260807 03:12:23.635128 1844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea41e668f055400ca3b64b4201bd128c" candidate_uuid: "059b6926da8c42f6bc96c7e97a141363" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a813aa28bc4313883d12f358cdd313"
I20260807 03:12:23.635192 1564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea41e668f055400ca3b64b4201bd128c" candidate_uuid: "059b6926da8c42f6bc96c7e97a141363" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a772b69d794e7b87f94cc0af808b20"
I20260807 03:12:23.635344 1844 raft_consensus.cc:3037] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:12:23.635367 1564 raft_consensus.cc:3037] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20 [term 0 FOLLOWER]: Advancing to term 1
W20260807 03:12:23.636706 1615 tablet.cc:2369] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:12:23.638689 1844 raft_consensus.cc:2454] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059b6926da8c42f6bc96c7e97a141363 in term 1.
I20260807 03:12:23.638689 1564 raft_consensus.cc:2454] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059b6926da8c42f6bc96c7e97a141363 in term 1.
I20260807 03:12:23.639129 1637 leader_election.cc:302] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059b6926da8c42f6bc96c7e97a141363, d6a772b69d794e7b87f94cc0af808b20; no voters:
I20260807 03:12:23.639547 1918 raft_consensus.cc:2781] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:12:23.640018 1918 raft_consensus.cc:686] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [term 1 LEADER]: Becoming Leader. State: Replica: 059b6926da8c42f6bc96c7e97a141363, State: Running, Role: LEADER
I20260807 03:12:23.640506 1918 consensus_queue.cc:237] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } } peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } }
I20260807 03:12:23.644753 1411 catalog_manager.cc:5495] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059b6926da8c42f6bc96c7e97a141363 (127.17.31.2). New cstate: current_term: 1 leader_uuid: "059b6926da8c42f6bc96c7e97a141363" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } health_report { overall_health: UNKNOWN } } }
I20260807 03:12:23.669317 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:23.663808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.17.31.2:34983)
0807 03:12:23.664018 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:23.664024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:23.664041 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:12:23.664264 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:23.664268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:23.664350 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:23.664499 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:12:23.664915 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.665630 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.667157 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.667567 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.667746 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:23.668215 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:12:23.668684 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:12:23.668842 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:23.668964 (+ 122us) server_negotiation.cc:299] Negotiation successful
0807 03:12:23.669056 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 03:12:23.677106 1844 raft_consensus.cc:1261] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313 [term 1 FOLLOWER]: Refusing update from remote peer 059b6926da8c42f6bc96c7e97a141363: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:12:23.677104 1564 raft_consensus.cc:1261] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Refusing update from remote peer 059b6926da8c42f6bc96c7e97a141363: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:12:23.678128 1922 consensus_queue.cc:1035] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:12:23.678558 1918 consensus_queue.cc:1035] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 03:12:23.689896 1755 tablet.cc:2369] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:12:23.706236 1412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:12:23.706578 1412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:12:23.710182 1412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Sending DeleteTablet for 3 replicas of tablet ea41e668f055400ca3b64b4201bd128c
I20260807 03:12:23.711397 1824 tablet_service.cc:1514] Processing DeleteTablet for tablet ea41e668f055400ca3b64b4201bd128c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:12:23 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:42494
I20260807 03:12:23.711506 1543 tablet_service.cc:1514] Processing DeleteTablet for tablet ea41e668f055400ca3b64b4201bd128c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:12:23 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:37416
I20260807 03:12:23.711671 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 1478
I20260807 03:12:23.712153 1685 tablet_service.cc:1514] Processing DeleteTablet for tablet ea41e668f055400ca3b64b4201bd128c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:12:23 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:38132
I20260807 03:12:23.712303 1935 tablet_replica.cc:330] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313: stopping tablet replica
I20260807 03:12:23.712721 1935 raft_consensus.cc:2229] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:12:23.712999 1937 tablet_replica.cc:330] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363: stopping tablet replica
I20260807 03:12:23.713327 1935 raft_consensus.cc:2258] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:12:23.713413 1937 raft_consensus.cc:2229] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:12:23.714097 1937 raft_consensus.cc:2258] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:12:23.715577 1935 ts_tablet_manager.cc:1905] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:12:23.715914 1937 ts_tablet_manager.cc:1905] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:12:23.722673 1935 ts_tablet_manager.cc:1918] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:12:23.722913 1935 log.cc:1196] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/ea41e668f055400ca3b64b4201bd128c
I20260807 03:12:23.723253 1937 ts_tablet_manager.cc:1918] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:12:23.723450 1937 log.cc:1196] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/ea41e668f055400ca3b64b4201bd128c
I20260807 03:12:23.723655 1935 ts_tablet_manager.cc:1939] T ea41e668f055400ca3b64b4201bd128c P 65a813aa28bc4313883d12f358cdd313: Deleting consensus metadata
I20260807 03:12:23.724030 1937 ts_tablet_manager.cc:1939] T ea41e668f055400ca3b64b4201bd128c P 059b6926da8c42f6bc96c7e97a141363: Deleting consensus metadata
I20260807 03:12:23.725711 1395 catalog_manager.cc:4841] TS 65a813aa28bc4313883d12f358cdd313 (127.17.31.3:36549): tablet ea41e668f055400ca3b64b4201bd128c (table test-table [id=dc5992ae492745a78f1d20e7a5fc2c9d]) successfully deleted
I20260807 03:12:23.725706 1396 catalog_manager.cc:4841] TS 059b6926da8c42f6bc96c7e97a141363 (127.17.31.2:34983): tablet ea41e668f055400ca3b64b4201bd128c (table test-table [id=dc5992ae492745a78f1d20e7a5fc2c9d]) successfully deleted
I20260807 03:12:23.726876 1936 tablet_replica.cc:330] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20: stopping tablet replica
I20260807 03:12:23.727401 1936 raft_consensus.cc:2229] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:12:23.727965 1936 raft_consensus.cc:2258] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:12:23.730033 1936 ts_tablet_manager.cc:1905] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 03:12:23 dist-test-slave-tm1g krb5kdc[1366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:12:23.737272 1936 ts_tablet_manager.cc:1918] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:12:23.737486 1936 log.cc:1196] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/ea41e668f055400ca3b64b4201bd128c
I20260807 03:12:23.738055 1936 ts_tablet_manager.cc:1939] T ea41e668f055400ca3b64b4201bd128c P d6a772b69d794e7b87f94cc0af808b20: Deleting consensus metadata
I20260807 03:12:23.740010 1396 catalog_manager.cc:4841] TS d6a772b69d794e7b87f94cc0af808b20 (127.17.31.1:42549): tablet ea41e668f055400ca3b64b4201bd128c (table test-table [id=dc5992ae492745a78f1d20e7a5fc2c9d]) successfully deleted
I20260807 03:12:23.746692 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:23.712473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.17.31.1:42549)
0807 03:12:23.712728 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:23.712737 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:23.723446 (+ 10709us) server_negotiation.cc:407] Connection header received
0807 03:12:23.723834 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:23.723843 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:23.723950 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:23.724032 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:23.726776 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.727601 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.729197 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.729643 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.729780 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:23.734222 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:23.734233 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:23.734245 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:23.734374 (+ 129us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:23.737905 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:23.738815 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:23.738824 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:23.738828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:23.738990 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:23.739412 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:23.739417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:23.739420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:23.744549 (+ 5129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:23.744933 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:23.745832 (+ 899us) server_negotiation.cc:299] Negotiation successful
0807 03:12:23.746131 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260807 03:12:23.748220 1608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:12:23.841372 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 1478
I20260807 03:12:23.869333 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 1620
Aug 07 03:12:23 dist-test-slave-tm1g krb5kdc[1366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:12:23.887241 1907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:23.870173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.17.31.2:34983)
0807 03:12:23.870414 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:23.870421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:23.870463 (+ 42us) server_negotiation.cc:407] Connection header received
0807 03:12:23.870663 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:23.870669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:23.870760 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:23.870896 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:23.872127 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.873494 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.875007 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:23.875708 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:23.875856 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:23.879973 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:23.879988 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:23.880009 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:23.880056 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:23.884232 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:23.884975 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:23.884985 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:23.884989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:23.885145 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:23.885550 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:23.885555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:23.885558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:23.885929 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:23.886284 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:23.886621 (+ 337us) server_negotiation.cc:299] Negotiation successful
0807 03:12:23.886844 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 03:12:23.887969 1749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:12:24.003176 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 1620
I20260807 03:12:24.024996 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 1761
Aug 07 03:12:24 dist-test-slave-tm1g krb5kdc[1366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:12:24.039659 1908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:24.025769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42500 (local address 127.17.31.3:36549)
0807 03:12:24.025959 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:24.025963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:24.025995 (+ 32us) server_negotiation.cc:407] Connection header received
0807 03:12:24.026139 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:24.026143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:24.026214 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:24.026290 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:24.027584 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:24.028289 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:24.029548 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:24.029946 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:24.030037 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:24.034668 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:24.034681 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:24.034697 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:24.034729 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:24.037256 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:24.037935 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:24.037941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:24.037943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:24.038079 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:24.038470 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:24.038473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:24.038475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:24.038812 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:24.039020 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:24.039262 (+ 242us) server_negotiation.cc:299] Negotiation successful
0807 03:12:24.039420 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260807 03:12:24.040349 1887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:12:24.129385 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 1761
I20260807 03:12:24.147643 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 1379
I20260807 03:12:24.148803 1444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:12:24.256824 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 1379
I20260807 03:12:44.279402 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:40745
--webserver_interface=127.17.31.62
--webserver_port=39643
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:40745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:12:44.598811 1944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:12:44.599113 1944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:12:44.599264 1944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:12:44.606199 1944 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 03:12:44.606305 1944 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 03:12:44.606405 1944 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 03:12:44.606493 1944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:12:44.606585 1944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:12:44.606698 1944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:12:44.606776 1944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:12:44.606854 1944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:12:44.616206 1944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:40745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:40745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=39643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:12:44.617406 1944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:12:44.618692 1944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:12:44.640540 1951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:44.640590 1950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:44.641294 1953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:12:44.641319 1944 server_base.cc:1034] running on GCE node
I20260807 03:12:44.642169 1944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:12:44.643761 1944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:12:44.645026 1944 hybrid_clock.cc:648] HybridClock initialized: now 1786072364644965 us; error 47 us; skew 500 ppm
I20260807 03:12:44.645411 1944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:12:44 dist-test-slave-tm1g krb5kdc[1366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072364, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:12:44.653187 1944 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:12:44.654819 1944 webserver.cc:460] Webserver started at http://127.17.31.62:39643/ using document root <none> and password file <none>
I20260807 03:12:44.655622 1944 fs_manager.cc:359] Metadata directory not provided
I20260807 03:12:44.655771 1944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:12:44.661592 1944 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260807 03:12:44.664611 1961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:44.665969 1944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:12:44.666172 1944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5"
format_stamp: "Formatted at 2026-08-07 03:12:21 on dist-test-slave-tm1g"
server_key: "98e20d169a6ae0a6cda855650e29bb02"
server_key_iv: "7cfab152effebceb1788f88e4797c43f"
server_key_version: "encryptionkey@0"
I20260807 03:12:44.667245 1944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:12:44.700500 1944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:12:44.704556 1944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:12:44.704857 1944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:12:44.723702 1944 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:40745
I20260807 03:12:44.723732 2013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:40745 every 8 connection(s)
I20260807 03:12:44.725216 1944 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:12:44.731994 2014 sys_catalog.cc:263] Verifying existing consensus state
I20260807 03:12:44.735074 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 1944
I20260807 03:12:44.735613 2014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Bootstrap starting.
I20260807 03:12:44.736142 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:42549
--local_ip_for_outbound_sockets=127.17.31.1
--tserver_master_addrs=127.17.31.62:40745
--webserver_port=36277
--webserver_interface=127.17.31.1
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:12:44.758471 2014 log.cc:824] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Log is configured to *not* fsync() on all Append() calls
I20260807 03:12:44.770385 2014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:12:44.770982 2014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Bootstrap complete.
I20260807 03:12:44.780215 2014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } }
I20260807 03:12:44.781059 2014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a53acbae1f24a0e97f0a0e4a7b884f5, State: Initialized, Role: FOLLOWER
I20260807 03:12:44.781669 2014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } }
I20260807 03:12:44.781919 2014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:12:44.782042 2014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:12:44.782236 2014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 1 FOLLOWER]: Advancing to term 2
I20260807 03:12:44.785915 2014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } }
I20260807 03:12:44.786525 2014 leader_election.cc:302] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a53acbae1f24a0e97f0a0e4a7b884f5; no voters:
I20260807 03:12:44.787425 2014 leader_election.cc:288] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 03:12:44.787724 2018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 03:12:44.790335 2018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 2 LEADER]: Becoming Leader. State: Replica: 4a53acbae1f24a0e97f0a0e4a7b884f5, State: Running, Role: LEADER
I20260807 03:12:44.790942 2018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } }
I20260807 03:12:44.791993 2014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:12:44.793776 2019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a53acbae1f24a0e97f0a0e4a7b884f5. Latest consensus state: current_term: 2 leader_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } } }
I20260807 03:12:44.794327 2019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [sys.catalog]: This master's current role is: LEADER
I20260807 03:12:44.794843 2020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } } }
I20260807 03:12:44.795238 2020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [sys.catalog]: This master's current role is: LEADER
I20260807 03:12:44.801064 2023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:12:44.810040 2023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:12:44.811571 2023 catalog_manager.cc:1259] Loaded cluster ID: b7b3db2bd55248b39d55a906a1aadc35
I20260807 03:12:44.811718 2023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:12:44.815807 2023 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:12:44.817391 2023 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Loaded TSK: 0
I20260807 03:12:44.849454 2023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Generated new TSK 1
I20260807 03:12:44.849833 2023 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 03:12:45.098774 2016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:12:45.099059 2016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:12:45.099246 2016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:12:45.106595 2016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:12:45.106725 2016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:12:45.106941 2016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:12:45.116310 2016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:42549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=36277
--tserver_master_addrs=127.17.31.62:40745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:12:45.117607 2016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:12:45.119006 2016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:12:45.131750 2042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:45.131996 2044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:45.131748 2041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:12:45.132692 2016 server_base.cc:1034] running on GCE node
I20260807 03:12:45.133280 2016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:12:45.134228 2016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:12:45.135454 2016 hybrid_clock.cc:648] HybridClock initialized: now 1786072365135399 us; error 49 us; skew 500 ppm
I20260807 03:12:45.135777 2016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:12:45 dist-test-slave-tm1g krb5kdc[1366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072365, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:12:45.139246 2016 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:12:45.140901 2016 webserver.cc:460] Webserver started at http://127.17.31.1:36277/ using document root <none> and password file <none>
I20260807 03:12:45.141716 2016 fs_manager.cc:359] Metadata directory not provided
I20260807 03:12:45.141888 2016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:12:45.148777 2016 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 03:12:45.151896 2052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:45.153383 2016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 03:12:45.153539 2016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "d6a772b69d794e7b87f94cc0af808b20"
format_stamp: "Formatted at 2026-08-07 03:12:22 on dist-test-slave-tm1g"
server_key: "bea1f651ca29b029ac52866a3bccfe5f"
server_key_iv: "af02a5e05b7a2700e11327014b2b0573"
server_key_version: "encryptionkey@0"
I20260807 03:12:45.154788 2016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:12:45.181187 2016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:12:45.185216 2016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:12:45.185473 2016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:12:45.186372 2016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:12:45.187600 2016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:12:45.187685 2016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:45.187759 2016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:12:45.187813 2016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:45.219638 2016 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:42549
I20260807 03:12:45.219741 2169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:42549 every 8 connection(s)
I20260807 03:12:45.221002 2016 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:12:45.226863 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 2016
I20260807 03:12:45.227489 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:34983
--local_ip_for_outbound_sockets=127.17.31.2
--tserver_master_addrs=127.17.31.62:40745
--webserver_port=41107
--webserver_interface=127.17.31.2
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 03:12:45 dist-test-slave-tm1g krb5kdc[1366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072365, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:12:45.244386 2173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:45.224048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:33639)
0807 03:12:45.224941 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:45.224992 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:45.226831 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:45.228253 (+ 1422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:45.228264 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:45.228873 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:12:45.229394 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:45.229411 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:45.231082 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:45.231086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:45.232276 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:45.232289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:45.233324 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:45.233331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:45.233600 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:45.234283 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:12:45.234356 (+ 73us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:12:45.236815 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:12:45.240208 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:45.240216 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:45.240229 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:45.240623 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:45.241062 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:45.241066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:45.241070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:45.241242 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:45.241953 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:12:45.241962 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:12:45.242285 (+ 323us) client_negotiation.cc:769] Sending connection context
0807 03:12:45.242604 (+ 319us) client_negotiation.cc:240] Negotiation successful
0807 03:12:45.243120 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":391,"threads_started":1}
I20260807 03:12:45.244405 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:45.223818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:33639 (local address 127.17.31.62:40745)
0807 03:12:45.224347 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:45.224358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:45.225120 (+ 762us) server_negotiation.cc:407] Connection header received
0807 03:12:45.227364 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:45.227375 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:45.227748 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:45.228106 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:45.229594 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:45.230898 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:45.232432 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:45.233096 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:45.233289 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:45.237071 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:45.237079 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:45.237090 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:45.237129 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:45.240063 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:45.240775 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:45.240782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:45.240787 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:45.240940 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:45.241349 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:45.241356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:45.241359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:45.241812 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:45.242015 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:45.242822 (+ 807us) server_negotiation.cc:299] Negotiation successful
0807 03:12:45.243190 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":227,"threads_started":1}
I20260807 03:12:45.247246 2170 heartbeater.cc:347] Connected to a master server at 127.17.31.62:40745
I20260807 03:12:45.247612 2170 heartbeater.cc:464] Registering TS with master...
I20260807 03:12:45.248528 2170 heartbeater.cc:511] Master 127.17.31.62:40745 requested a full tablet report, sending...
I20260807 03:12:45.250831 1979 ts_manager.cc:194] Registered new tserver with Master: d6a772b69d794e7b87f94cc0af808b20 (127.17.31.1:42549)
I20260807 03:12:45.252297 1979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:33639
W20260807 03:12:45.558378 2174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:12:45.558660 2174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:12:45.558847 2174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:12:45.565850 2174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:12:45.565968 2174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:12:45.566145 2174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:12:45.575219 2174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:34983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=41107
--tserver_master_addrs=127.17.31.62:40745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:12:45.576401 2174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:12:45.577683 2174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:12:45.590186 2179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:45.590160 2182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:45.590116 2180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:12:45.591452 2174 server_base.cc:1034] running on GCE node
I20260807 03:12:45.592052 2174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:12:45.592892 2174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:12:45.594111 2174 hybrid_clock.cc:648] HybridClock initialized: now 1786072365594050 us; error 68 us; skew 500 ppm
I20260807 03:12:45.594472 2174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:12:45 dist-test-slave-tm1g krb5kdc[1366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072365, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:12:45.597597 2174 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:12:45.598980 2174 webserver.cc:460] Webserver started at http://127.17.31.2:41107/ using document root <none> and password file <none>
I20260807 03:12:45.599653 2174 fs_manager.cc:359] Metadata directory not provided
I20260807 03:12:45.599804 2174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:12:45.604888 2174 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 03:12:45.607430 2189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:45.608744 2174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:12:45.608911 2174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "059b6926da8c42f6bc96c7e97a141363"
format_stamp: "Formatted at 2026-08-07 03:12:22 on dist-test-slave-tm1g"
server_key: "6e36f5fc9379f5b40cf60550ff93416a"
server_key_iv: "9d2804ca423099098de6e6cd9803c751"
server_key_version: "encryptionkey@0"
I20260807 03:12:45.609825 2174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:12:45.622009 2174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:12:45.625968 2174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:12:45.626222 2174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:12:45.627122 2174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:12:45.628334 2174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:12:45.628451 2174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:45.628577 2174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:12:45.628662 2174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:45.660429 2174 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:34983
I20260807 03:12:45.660561 2307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:34983 every 8 connection(s)
I20260807 03:12:45.661774 2174 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:12:45.663964 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 2174
I20260807 03:12:45.664721 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:36549
--local_ip_for_outbound_sockets=127.17.31.3
--tserver_master_addrs=127.17.31.62:40745
--webserver_port=38191
--webserver_interface=127.17.31.3
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 03:12:45 dist-test-slave-tm1g krb5kdc[1366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072365, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:12:45.681355 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:45.664308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:48771 (local address 127.17.31.62:40745)
0807 03:12:45.664479 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:45.664483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:45.665822 (+ 1339us) server_negotiation.cc:407] Connection header received
0807 03:12:45.667652 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:45.667658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:45.667745 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:45.667851 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:45.669279 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:45.669988 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:45.671689 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:45.672180 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:45.672309 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:45.675808 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:45.675813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:45.675816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:45.675849 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:45.678440 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:45.679105 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:45.679111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:45.679114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:45.679238 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:45.679692 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:45.679697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:45.679701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:45.679977 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:45.680141 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:45.680775 (+ 634us) server_negotiation.cc:299] Negotiation successful
0807 03:12:45.681013 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 03:12:45.682232 2310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:45.664909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:48771)
0807 03:12:45.665667 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:45.665704 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:45.667391 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:45.668081 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:45.668089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:45.668576 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:12:45.669112 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:45.669129 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:45.670167 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:45.670171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:45.671549 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:45.671562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:45.672405 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:45.672413 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:45.672633 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:45.673297 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:12:45.673323 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:12:45.675569 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:12:45.678569 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:45.678577 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:45.678590 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:45.678976 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:45.679344 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:45.679348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:45.679352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:45.679570 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:45.680111 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:12:45.680119 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:12:45.680424 (+ 305us) client_negotiation.cc:769] Sending connection context
0807 03:12:45.680678 (+ 254us) client_negotiation.cc:240] Negotiation successful
0807 03:12:45.681170 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":320,"threads_started":1}
I20260807 03:12:45.683679 2308 heartbeater.cc:347] Connected to a master server at 127.17.31.62:40745
I20260807 03:12:45.683967 2308 heartbeater.cc:464] Registering TS with master...
I20260807 03:12:45.684652 2308 heartbeater.cc:511] Master 127.17.31.62:40745 requested a full tablet report, sending...
I20260807 03:12:45.686144 1979 ts_manager.cc:194] Registered new tserver with Master: 059b6926da8c42f6bc96c7e97a141363 (127.17.31.2:34983)
I20260807 03:12:45.687507 1979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:48771
W20260807 03:12:45.994160 2311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:12:45.994481 2311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:12:45.994588 2311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:12:46.001684 2311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:12:46.001780 2311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:12:46.001888 2311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:12:46.011128 2311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:36549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=38191
--tserver_master_addrs=127.17.31.62:40745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:12:46.012187 2311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:12:46.013397 2311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:12:46.025837 2319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:46.025883 2316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:12:46.025907 2317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:12:46.027073 2311 server_base.cc:1034] running on GCE node
I20260807 03:12:46.027597 2311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:12:46.028412 2311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:12:46.029590 2311 hybrid_clock.cc:648] HybridClock initialized: now 1786072366029556 us; error 29 us; skew 500 ppm
I20260807 03:12:46.029873 2311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:12:46 dist-test-slave-tm1g krb5kdc[1366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072366, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:12:46.032788 2311 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:12:46.034101 2311 webserver.cc:460] Webserver started at http://127.17.31.3:38191/ using document root <none> and password file <none>
I20260807 03:12:46.034795 2311 fs_manager.cc:359] Metadata directory not provided
I20260807 03:12:46.034935 2311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:12:46.039893 2311 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:12:46.042364 2326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:46.043538 2311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 03:12:46.043704 2311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "65a813aa28bc4313883d12f358cdd313"
format_stamp: "Formatted at 2026-08-07 03:12:23 on dist-test-slave-tm1g"
server_key: "1d4f9c6a3bd0742703b2014955c7d015"
server_key_iv: "98534258f4ee3cfdedbf00333722a11c"
server_key_version: "encryptionkey@0"
I20260807 03:12:46.044615 2311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:12:46.075205 2311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:12:46.079164 2311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:12:46.079417 2311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:12:46.080348 2311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:12:46.081585 2311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:12:46.081704 2311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:46.081830 2311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:12:46.081910 2311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:12:46.114846 2311 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:36549
I20260807 03:12:46.114926 2442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:36549 every 8 connection(s)
I20260807 03:12:46.116201 2311 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:12:46.120983 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 2311
Aug 07 03:12:46 dist-test-slave-tm1g krb5kdc[1366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072366, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:12:46.136315 2446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.121869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.17.31.62:40745)
0807 03:12:46.122349 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:46.122357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:46.122702 (+ 345us) server_negotiation.cc:407] Connection header received
0807 03:12:46.122887 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:46.122896 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:46.122944 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:46.123047 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:46.124887 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.125654 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.127206 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.127652 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.127787 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.130649 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:46.130655 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:46.130658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:46.130687 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:46.133605 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:46.134277 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:46.134283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:46.134290 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:46.134465 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:46.134844 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:46.134849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:46.134853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:46.135126 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:46.135412 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:46.135689 (+ 277us) server_negotiation.cc:299] Negotiation successful
0807 03:12:46.135900 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":238,"threads_started":1}
I20260807 03:12:46.136972 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.118697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:40935 (local address 127.17.31.62:40745)
0807 03:12:46.118907 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:46.118913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:46.120114 (+ 1201us) server_negotiation.cc:407] Connection header received
0807 03:12:46.122226 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:46.122235 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:46.122371 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:46.122447 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:46.124076 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.125157 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.126913 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.127372 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.127498 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.130856 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:46.130861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:46.130864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:46.130908 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:46.133963 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:46.134708 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:46.134715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:46.134718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:46.134836 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:46.135277 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:46.135282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:46.135290 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:46.135528 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:46.135685 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:46.136390 (+ 705us) server_negotiation.cc:299] Negotiation successful
0807 03:12:46.136592 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 03:12:46.137642 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.119258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:40935)
0807 03:12:46.119959 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:46.119996 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:46.121874 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:46.122870 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:46.122877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:46.123379 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:12:46.123904 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:46.123920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.125345 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.125349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:46.126779 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:46.126792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.127593 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.127603 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:46.127849 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.128521 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:12:46.128543 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:12:46.130663 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:12:46.134103 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:46.134111 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:46.134125 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:46.134593 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:46.134975 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:12:46.134979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:12:46.134983 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:12:46.135160 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:12:46.135661 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:12:46.135668 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:12:46.135963 (+ 295us) client_negotiation.cc:769] Sending connection context
0807 03:12:46.136249 (+ 286us) client_negotiation.cc:240] Negotiation successful
0807 03:12:46.136580 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":272,"threads_started":1}
I20260807 03:12:46.139045 2443 heartbeater.cc:347] Connected to a master server at 127.17.31.62:40745
I20260807 03:12:46.139323 2443 heartbeater.cc:464] Registering TS with master...
I20260807 03:12:46.139992 2443 heartbeater.cc:511] Master 127.17.31.62:40745 requested a full tablet report, sending...
I20260807 03:12:46.141271 1979 ts_manager.cc:194] Registered new tserver with Master: 65a813aa28bc4313883d12f358cdd313 (127.17.31.3:36549)
I20260807 03:12:46.142027 1979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:40935
I20260807 03:12:46.151253 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:12:46.157377 2446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.152131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.17.31.62:40745)
0807 03:12:46.152321 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:46.152330 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:46.152664 (+ 334us) server_negotiation.cc:407] Connection header received
0807 03:12:46.152840 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:46.152844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:46.152955 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:46.153066 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:12:46.153417 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.154104 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.155714 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.156127 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.156215 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.156643 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:12:46.156897 (+ 254us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0807 03:12:46.157139 (+ 242us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52874: token expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260807 03:12:46.157505 2446 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52874: token expired
W20260807 03:12:46.158044 1898 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.31.62:40745)
I20260807 03:12:46.169909 2446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.158507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.17.31.62:40745)
0807 03:12:46.158687 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:46.158694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:46.158896 (+ 202us) server_negotiation.cc:407] Connection header received
0807 03:12:46.158999 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:46.159003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:46.159052 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:46.159122 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:46.160570 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.161210 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.162520 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.162898 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.162982 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.165397 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:46.165402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:46.165405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:46.165430 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:46.167748 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:46.168341 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:46.168344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:46.168347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:46.168460 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:46.168784 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:46.168787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:46.168789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:46.169059 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:46.169236 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:46.169464 (+ 228us) server_negotiation.cc:299] Negotiation successful
0807 03:12:46.169608 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 03:12:46.173058 1979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:12:46.180071 1979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:12:46.201467 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.191400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:36549 (local address 127.0.0.1:46960)
0807 03:12:46.192347 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:46.192366 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:46.192476 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:46.193713 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:46.193721 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:46.193751 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:12:46.194168 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:46.194179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.196451 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.196456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:46.198711 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:46.198725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.200609 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.200616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:46.200770 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.200804 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 03:12:46.200887 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 03:12:46.201007 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":2,"thread_start_us":911,"threads_started":1}
I20260807 03:12:46.202190 2453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.190710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:42549 (local address 127.0.0.1:48592)
0807 03:12:46.191487 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:46.191533 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:46.191698 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:46.194497 (+ 2799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:46.194502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:46.194527 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:12:46.195118 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:46.195139 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.197125 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.197132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:46.199435 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:46.199450 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.201264 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.201271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:46.201511 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.201564 (+ 53us) client_negotiation.cc:769] Sending connection context
0807 03:12:46.201659 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 03:12:46.201774 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":104,"thread_start_us":564,"threads_started":1}
I20260807 03:12:46.202268 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.192036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.17.31.3:36549)
0807 03:12:46.193076 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:46.193086 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:46.193122 (+ 36us) server_negotiation.cc:407] Connection header received
0807 03:12:46.193220 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:46.193228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:46.193463 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:46.194417 (+ 954us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:12:46.194713 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.196302 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.198907 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.200377 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.201476 (+ 1099us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.201706 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:46.201773 (+ 67us) server_negotiation.cc:299] Negotiation successful
0807 03:12:46.201936 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":705,"threads_started":1}
I20260807 03:12:46.202426 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.192287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:34983 (local address 127.0.0.1:54588)
0807 03:12:46.193307 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:46.193323 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:46.193422 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:46.194870 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:46.194882 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:46.194908 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:12:46.195434 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:46.195443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.197067 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.197072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:46.199452 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:46.199464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.201568 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.201574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:46.201751 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.201782 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 03:12:46.201869 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 03:12:46.201986 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":113,"thread_start_us":875,"threads_started":1}
I20260807 03:12:46.202970 2454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.190988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.17.31.1:42549)
0807 03:12:46.193919 (+ 2931us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:46.193929 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:46.193957 (+ 28us) server_negotiation.cc:407] Connection header received
0807 03:12:46.194048 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:46.194058 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:46.194281 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:46.194490 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:12:46.195334 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.196958 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.199602 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.201051 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.201170 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.201356 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:46.202573 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0807 03:12:46.202739 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2762,"thread_start_us":2643,"threads_started":1}
I20260807 03:12:46.203315 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.192948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54588 (local address 127.17.31.2:34983)
0807 03:12:46.194200 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:46.194209 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:46.194236 (+ 27us) server_negotiation.cc:407] Connection header received
0807 03:12:46.194401 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:46.194407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:46.194642 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:46.194853 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:12:46.195578 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.196892 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.199643 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.201043 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.201170 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.201356 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:46.202827 (+ 1471us) server_negotiation.cc:299] Negotiation successful
0807 03:12:46.202974 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1063,"thread_start_us":1168,"threads_started":1}
I20260807 03:12:46.205909 2376 tablet_service.cc:1467] Processing CreateTablet for tablet 8cdcf3c52ec54cd78de6138f5298619b (DEFAULT_TABLE table=test-table [id=724dd8aaef7442fea61b545c24502754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:12:46.206094 2103 tablet_service.cc:1467] Processing CreateTablet for tablet 8cdcf3c52ec54cd78de6138f5298619b (DEFAULT_TABLE table=test-table [id=724dd8aaef7442fea61b545c24502754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:12:46.207295 2376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cdcf3c52ec54cd78de6138f5298619b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:12:46.207345 2103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cdcf3c52ec54cd78de6138f5298619b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:12:46.209626 2241 tablet_service.cc:1467] Processing CreateTablet for tablet 8cdcf3c52ec54cd78de6138f5298619b (DEFAULT_TABLE table=test-table [id=724dd8aaef7442fea61b545c24502754]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:12:46.211172 2241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cdcf3c52ec54cd78de6138f5298619b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:12:46.219188 2459 tablet_bootstrap.cc:492] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20: Bootstrap starting.
I20260807 03:12:46.223512 2459 tablet_bootstrap.cc:654] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20: Neither blocks nor log segments found. Creating new log.
I20260807 03:12:46.225286 2459 log.cc:824] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20: Log is configured to *not* fsync() on all Append() calls
I20260807 03:12:46.233428 2462 tablet_bootstrap.cc:492] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363: Bootstrap starting.
I20260807 03:12:46.235984 2460 tablet_bootstrap.cc:492] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313: Bootstrap starting.
I20260807 03:12:46.237795 2459 tablet_bootstrap.cc:492] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20: No bootstrap required, opened a new log
I20260807 03:12:46.238095 2459 ts_tablet_manager.cc:1397] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.009s
I20260807 03:12:46.238312 2462 tablet_bootstrap.cc:654] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363: Neither blocks nor log segments found. Creating new log.
I20260807 03:12:46.239113 2460 tablet_bootstrap.cc:654] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313: Neither blocks nor log segments found. Creating new log.
I20260807 03:12:46.239562 2462 log.cc:824] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363: Log is configured to *not* fsync() on all Append() calls
I20260807 03:12:46.240062 2460 log.cc:824] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313: Log is configured to *not* fsync() on all Append() calls
I20260807 03:12:46.242487 2462 tablet_bootstrap.cc:492] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363: No bootstrap required, opened a new log
I20260807 03:12:46.242739 2462 ts_tablet_manager.cc:1397] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260807 03:12:46.243257 2460 tablet_bootstrap.cc:492] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313: No bootstrap required, opened a new log
I20260807 03:12:46.243486 2460 ts_tablet_manager.cc:1397] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260807 03:12:46.248636 2459 raft_consensus.cc:348] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } } peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } }
I20260807 03:12:46.249107 2459 raft_consensus.cc:374] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:12:46.249264 2459 raft_consensus.cc:729] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a772b69d794e7b87f94cc0af808b20, State: Initialized, Role: FOLLOWER
I20260807 03:12:46.249857 2459 consensus_queue.cc:260] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } } peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } }
I20260807 03:12:46.250739 2460 raft_consensus.cc:348] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } } peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } }
I20260807 03:12:46.251052 2460 raft_consensus.cc:374] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:12:46.251157 2460 raft_consensus.cc:729] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a813aa28bc4313883d12f358cdd313, State: Initialized, Role: FOLLOWER
I20260807 03:12:46.251652 2460 consensus_queue.cc:260] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } } peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } }
I20260807 03:12:46.252625 2443 heartbeater.cc:503] Master 127.17.31.62:40745 was elected leader, sending a full tablet report...
I20260807 03:12:46.253233 2460 ts_tablet_manager.cc:1428] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260807 03:12:46.254115 2462 raft_consensus.cc:348] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } } peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } }
I20260807 03:12:46.254664 2462 raft_consensus.cc:374] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:12:46.254797 2462 raft_consensus.cc:729] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059b6926da8c42f6bc96c7e97a141363, State: Initialized, Role: FOLLOWER
I20260807 03:12:46.255463 2462 consensus_queue.cc:260] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } } peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } }
I20260807 03:12:46.255960 2170 heartbeater.cc:503] Master 127.17.31.62:40745 was elected leader, sending a full tablet report...
I20260807 03:12:46.257790 2459 ts_tablet_manager.cc:1428] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20: Time spent starting tablet: real 0.020s user 0.011s sys 0.009s
I20260807 03:12:46.260167 2308 heartbeater.cc:503] Master 127.17.31.62:40745 was elected leader, sending a full tablet report...
I20260807 03:12:46.261015 2462 ts_tablet_manager.cc:1428] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363: Time spent starting tablet: real 0.018s user 0.010s sys 0.006s
W20260807 03:12:46.367883 2444 tablet.cc:2369] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:12:46.413692 2309 tablet.cc:2369] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:12:46.473227 2171 tablet.cc:2369] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:12:46.492512 2466 raft_consensus.cc:482] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:12:46.492801 2466 raft_consensus.cc:504] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } } peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } }
I20260807 03:12:46.494546 2466 leader_election.cc:288] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a772b69d794e7b87f94cc0af808b20 (127.17.31.1:42549), 059b6926da8c42f6bc96c7e97a141363 (127.17.31.2:34983)
I20260807 03:12:46.500584 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.494939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:42549 (local address 127.17.31.3:35461)
0807 03:12:46.495130 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:46.495150 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:46.495250 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:46.495644 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:46.495648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:46.495677 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:12:46.495965 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:46.495984 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.497118 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.497123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:46.498887 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:46.498901 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.500049 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.500053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:46.500186 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.500208 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 03:12:46.500295 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 03:12:46.500382 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 03:12:46.500735 2454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.494918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:35461 (local address 127.17.31.1:42549)
0807 03:12:46.495155 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:46.495161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:46.495207 (+ 46us) server_negotiation.cc:407] Connection header received
0807 03:12:46.495411 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:46.495416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:46.495506 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:46.495616 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:12:46.496139 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.496984 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.499066 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.499904 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.500042 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.500146 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:46.500355 (+ 209us) server_negotiation.cc:299] Negotiation successful
0807 03:12:46.500493 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 03:12:46.501904 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.495640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:47323 (local address 127.17.31.2:34983)
0807 03:12:46.495957 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:46.495963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:46.496312 (+ 349us) server_negotiation.cc:407] Connection header received
0807 03:12:46.496506 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:46.496510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:46.496571 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:46.496672 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:12:46.497272 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.497956 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.499939 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.501110 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.501246 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.501309 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:46.501653 (+ 344us) server_negotiation.cc:299] Negotiation successful
0807 03:12:46.501736 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260807 03:12:46.501684 2123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cdcf3c52ec54cd78de6138f5298619b" candidate_uuid: "65a813aa28bc4313883d12f358cdd313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a772b69d794e7b87f94cc0af808b20" is_pre_election: true
I20260807 03:12:46.502104 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.495419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:34983 (local address 127.17.31.3:47323)
0807 03:12:46.496222 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:46.496234 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 03:12:46.496354 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:12:46.496713 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:12:46.496717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:12:46.496739 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:12:46.497109 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:46.497117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.498097 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.498103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:46.499798 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:12:46.499808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.501324 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.501330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:12:46.501500 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.501535 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 03:12:46.501636 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 03:12:46.501766 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":495,"threads_started":1}
I20260807 03:12:46.502278 2123 raft_consensus.cc:2454] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65a813aa28bc4313883d12f358cdd313 in term 0.
I20260807 03:12:46.502930 2328 leader_election.cc:302] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65a813aa28bc4313883d12f358cdd313, d6a772b69d794e7b87f94cc0af808b20; no voters:
I20260807 03:12:46.503091 2262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cdcf3c52ec54cd78de6138f5298619b" candidate_uuid: "65a813aa28bc4313883d12f358cdd313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059b6926da8c42f6bc96c7e97a141363" is_pre_election: true
I20260807 03:12:46.503325 2466 raft_consensus.cc:2781] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:12:46.503494 2466 raft_consensus.cc:482] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:12:46.503615 2466 raft_consensus.cc:3037] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:12:46.503675 2262 raft_consensus.cc:2454] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65a813aa28bc4313883d12f358cdd313 in term 0.
I20260807 03:12:46.505934 2466 raft_consensus.cc:504] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } } peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } }
I20260807 03:12:46.506500 2466 leader_election.cc:288] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [CANDIDATE]: Term 1 election: Requested vote from peers d6a772b69d794e7b87f94cc0af808b20 (127.17.31.1:42549), 059b6926da8c42f6bc96c7e97a141363 (127.17.31.2:34983)
I20260807 03:12:46.506877 2123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cdcf3c52ec54cd78de6138f5298619b" candidate_uuid: "65a813aa28bc4313883d12f358cdd313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a772b69d794e7b87f94cc0af808b20"
I20260807 03:12:46.506917 2262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cdcf3c52ec54cd78de6138f5298619b" candidate_uuid: "65a813aa28bc4313883d12f358cdd313" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059b6926da8c42f6bc96c7e97a141363"
I20260807 03:12:46.507098 2123 raft_consensus.cc:3037] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:12:46.507103 2262 raft_consensus.cc:3037] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:12:46.510229 2123 raft_consensus.cc:2454] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65a813aa28bc4313883d12f358cdd313 in term 1.
I20260807 03:12:46.510229 2262 raft_consensus.cc:2454] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65a813aa28bc4313883d12f358cdd313 in term 1.
I20260807 03:12:46.510597 2328 leader_election.cc:302] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65a813aa28bc4313883d12f358cdd313, d6a772b69d794e7b87f94cc0af808b20; no voters:
I20260807 03:12:46.510835 2466 raft_consensus.cc:2781] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:12:46.511265 2466 raft_consensus.cc:686] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [term 1 LEADER]: Becoming Leader. State: Replica: 65a813aa28bc4313883d12f358cdd313, State: Running, Role: LEADER
I20260807 03:12:46.511710 2466 consensus_queue.cc:237] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } } peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } }
I20260807 03:12:46.516258 1978 catalog_manager.cc:5495] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65a813aa28bc4313883d12f358cdd313 (127.17.31.3). New cstate: current_term: 1 leader_uuid: "65a813aa28bc4313883d12f358cdd313" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } health_report { overall_health: HEALTHY } } }
I20260807 03:12:46.566506 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.561046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.17.31.3:36549)
0807 03:12:46.561240 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:46.561246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:46.561408 (+ 162us) server_negotiation.cc:407] Connection header received
0807 03:12:46.561639 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:46.561643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:46.561714 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:46.561861 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:12:46.562240 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.562982 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.564429 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.564829 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.564956 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.565446 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:12:46.565919 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:12:46.566091 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:46.566150 (+ 59us) server_negotiation.cc:299] Negotiation successful
0807 03:12:46.566224 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 03:12:46.573904 2262 raft_consensus.cc:1261] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363 [term 1 FOLLOWER]: Refusing update from remote peer 65a813aa28bc4313883d12f358cdd313: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:12:46.573909 2123 raft_consensus.cc:1261] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Refusing update from remote peer 65a813aa28bc4313883d12f358cdd313: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:12:46.574757 2469 consensus_queue.cc:1035] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:12:46.575098 2466 consensus_queue.cc:1035] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [LEADER]: Connected to new peer: Peer: permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:12:46.587162 2474 mvcc.cc:205] Tried to move back new op lower bound from 7315752413477933056 to 7315752413235605504. Current Snapshot: MvccSnapshot[applied={T|T < 7315752413477933056}]
I20260807 03:12:46.598836 1978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:12:46.599114 1978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:12:46.603431 1978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Sending DeleteTablet for 3 replicas of tablet 8cdcf3c52ec54cd78de6138f5298619b
I20260807 03:12:46.604498 2103 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cdcf3c52ec54cd78de6138f5298619b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:12:46 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:48592
I20260807 03:12:46.604643 2241 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cdcf3c52ec54cd78de6138f5298619b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:12:46 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:54588
I20260807 03:12:46.604866 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 2016
I20260807 03:12:46.610392 2376 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cdcf3c52ec54cd78de6138f5298619b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:12:46 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:46960
I20260807 03:12:46.619948 2483 tablet_replica.cc:330] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363: stopping tablet replica
I20260807 03:12:46.620328 2483 raft_consensus.cc:2229] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:12:46.620801 2483 raft_consensus.cc:2258] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:12:46.624338 2483 ts_tablet_manager.cc:1905] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:12:46.627113 2454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.606450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.17.31.1:42549)
0807 03:12:46.606610 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:46.606618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:46.606638 (+ 20us) server_negotiation.cc:407] Connection header received
0807 03:12:46.606691 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:46.606695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:46.606797 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:46.606892 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:46.608815 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.610561 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.613160 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.614112 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.614260 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.618459 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:46.618472 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:46.618488 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:46.618531 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:46.622951 (+ 4420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:46.624046 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:46.624056 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:46.624061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:46.624226 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:46.624703 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:46.624707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:46.624711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:46.625207 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:46.625501 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:46.626488 (+ 987us) server_negotiation.cc:299] Negotiation successful
0807 03:12:46.626707 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 03:12:46.628504 2165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:12:46.629686 2484 tablet_replica.cc:330] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313: stopping tablet replica
I20260807 03:12:46.630208 2484 raft_consensus.cc:2229] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:12:46.630930 2484 raft_consensus.cc:2258] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:12:46.632835 2484 ts_tablet_manager.cc:1905] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:12:46.633127 2483 ts_tablet_manager.cc:1918] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:12:46.633347 2483 log.cc:1196] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/8cdcf3c52ec54cd78de6138f5298619b
I20260807 03:12:46.633984 2483 ts_tablet_manager.cc:1939] T 8cdcf3c52ec54cd78de6138f5298619b P 059b6926da8c42f6bc96c7e97a141363: Deleting consensus metadata
I20260807 03:12:46.636049 1963 catalog_manager.cc:4841] TS 059b6926da8c42f6bc96c7e97a141363 (127.17.31.2:34983): tablet 8cdcf3c52ec54cd78de6138f5298619b (table test-table [id=724dd8aaef7442fea61b545c24502754]) successfully deleted
I20260807 03:12:46.641269 2484 ts_tablet_manager.cc:1918] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:12:46.641440 2484 log.cc:1196] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/8cdcf3c52ec54cd78de6138f5298619b
I20260807 03:12:46.641979 2484 ts_tablet_manager.cc:1939] T 8cdcf3c52ec54cd78de6138f5298619b P 65a813aa28bc4313883d12f358cdd313: Deleting consensus metadata
I20260807 03:12:46.643738 1962 catalog_manager.cc:4841] TS 65a813aa28bc4313883d12f358cdd313 (127.17.31.3:36549): tablet 8cdcf3c52ec54cd78de6138f5298619b (table test-table [id=724dd8aaef7442fea61b545c24502754]) successfully deleted
I20260807 03:12:46.736405 2482 tablet_replica.cc:330] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20: stopping tablet replica
I20260807 03:12:46.737211 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 2016
I20260807 03:12:46.757069 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 2174
W20260807 03:12:46.757025 1963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.31.1:42549 (error 108)
I20260807 03:12:46.763125 2487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.759734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:42549 (local address 127.0.0.1:48612)
0807 03:12:46.762770 (+ 3036us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:46.762909 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2894,"thread_start_us":2803,"threads_started":1}
W20260807 03:12:46.763453 1963 catalog_manager.cc:4584] TS d6a772b69d794e7b87f94cc0af808b20 (127.17.31.1:42549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8cdcf3c52ec54cd78de6138f5298619b: Network error: Client connection negotiation failed: client connection to 127.17.31.1:42549: connect: Connection refused (error 111)
I20260807 03:12:46.774189 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.758243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.17.31.2:34983)
0807 03:12:46.758449 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:46.758456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:46.758486 (+ 30us) server_negotiation.cc:407] Connection header received
0807 03:12:46.758558 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:46.758563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:46.758662 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:46.758773 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:46.761023 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.762239 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.763997 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.764642 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.764773 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.767316 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:46.767336 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:46.767356 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:46.767398 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:46.771202 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:46.772020 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:46.772027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:46.772030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:46.772168 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:46.772562 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:46.772572 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:46.772574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:46.772961 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:46.773284 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:46.773609 (+ 325us) server_negotiation.cc:299] Negotiation successful
0807 03:12:46.773841 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 03:12:46.774865 2303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:12:46.817435 2487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.816852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:42549 (local address 127.0.0.1:48626)
0807 03:12:46.817114 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:46.817242 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260807 03:12:46.818770 2487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.818273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:42549 (local address 127.0.0.1:48640)
0807 03:12:46.818484 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:46.818585 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260807 03:12:46.867538 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 2174
I20260807 03:12:46.879508 2487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.879045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:42549 (local address 127.0.0.1:48656)
0807 03:12:46.879258 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:46.879355 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260807 03:12:46.880595 2487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.880239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:42549 (local address 127.0.0.1:48664)
0807 03:12:46.880384 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:46.880459 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 03:12:46.894404 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 2311
I20260807 03:12:46.914594 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.895258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46986 (local address 127.17.31.3:36549)
0807 03:12:46.895468 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 03:12:46.895476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:12:46.895511 (+ 35us) server_negotiation.cc:407] Connection header received
0807 03:12:46.895672 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:12:46.895678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:12:46.895779 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:12:46.895915 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:12:46.897361 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.898757 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.900281 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:12:46.900993 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:12:46.901138 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:12:46.907460 (+ 6322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:12:46.907476 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:12:46.907503 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:12:46.907557 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:12:46.911643 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:46.912293 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:46.912302 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:46.912306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:46.912513 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:12:46.912864 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:12:46.912869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:12:46.912872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:12:46.913373 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:12:46.913710 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0807 03:12:46.913917 (+ 207us) server_negotiation.cc:299] Negotiation successful
0807 03:12:46.914155 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 03:12:46.915390 2438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:12:46.961694 2487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.961108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:42549 (local address 127.0.0.1:48672)
0807 03:12:46.961380 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:46.961497 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 03:12:46.963606 2487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:12:46.963175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:42549 (local address 127.0.0.1:48678)
0807 03:12:46.963370 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 03:12:46.963450 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260807 03:12:47.051651 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 2311
I20260807 03:12:47.070161 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 1944
I20260807 03:12:47.071352 2009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:12:47.184150 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 1944
I20260807 03:13:07.207579 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:40745
--webserver_interface=127.17.31.62
--webserver_port=39643
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:40745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:07.535485 2491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:07.535765 2491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:07.535928 2491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:07.542927 2491 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 03:13:07.543011 2491 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 03:13:07.543104 2491 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 03:13:07.543208 2491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:13:07.543298 2491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:07.543419 2491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:07.543498 2491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:13:07.543587 2491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:13:07.553088 2491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:40745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:40745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=39643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:07.554370 2491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:07.555727 2491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:07.571419 2496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:07.571473 2499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:07.571499 2497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:07.572276 2491 server_base.cc:1034] running on GCE node
I20260807 03:13:07.573125 2491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:07.574631 2491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:07.575877 2491 hybrid_clock.cc:648] HybridClock initialized: now 1786072387575815 us; error 38 us; skew 500 ppm
I20260807 03:13:07.576207 2491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:07 dist-test-slave-tm1g krb5kdc[1366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072387, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:07.579309 2491 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:13:07.580809 2491 webserver.cc:460] Webserver started at http://127.17.31.62:39643/ using document root <none> and password file <none>
I20260807 03:13:07.581526 2491 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:07.581673 2491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:07.587154 2491 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:13:07.589892 2506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:07.591180 2491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:13:07.591367 2491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5"
format_stamp: "Formatted at 2026-08-07 03:12:21 on dist-test-slave-tm1g"
server_key: "98e20d169a6ae0a6cda855650e29bb02"
server_key_iv: "7cfab152effebceb1788f88e4797c43f"
server_key_version: "encryptionkey@0"
I20260807 03:13:07.592312 2491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:07.606797 2491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:07.610460 2491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:07.610743 2491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:07.629106 2491 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:40745
I20260807 03:13:07.629136 2558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:40745 every 8 connection(s)
I20260807 03:13:07.630674 2491 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:13:07.633595 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 2491
I20260807 03:13:07.634693 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:42549
--local_ip_for_outbound_sockets=127.17.31.1
--tserver_master_addrs=127.17.31.62:40745
--webserver_port=36277
--webserver_interface=127.17.31.1
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:13:07.639276 2559 sys_catalog.cc:263] Verifying existing consensus state
I20260807 03:13:07.644331 2559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Bootstrap starting.
I20260807 03:13:07.664695 2559 log.cc:824] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:07.680543 2559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:13:07.681082 2559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Bootstrap complete.
I20260807 03:13:07.690040 2559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } }
I20260807 03:13:07.690900 2559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a53acbae1f24a0e97f0a0e4a7b884f5, State: Initialized, Role: FOLLOWER
I20260807 03:13:07.691548 2559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } }
I20260807 03:13:07.691778 2559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:13:07.691905 2559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:13:07.692123 2559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 2 FOLLOWER]: Advancing to term 3
I20260807 03:13:07.696048 2559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } }
I20260807 03:13:07.696547 2559 leader_election.cc:302] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a53acbae1f24a0e97f0a0e4a7b884f5; no voters:
I20260807 03:13:07.697299 2559 leader_election.cc:288] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 03:13:07.697434 2564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 03:13:07.699594 2559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:13:07.702701 2564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [term 3 LEADER]: Becoming Leader. State: Replica: 4a53acbae1f24a0e97f0a0e4a7b884f5, State: Running, Role: LEADER
I20260807 03:13:07.703428 2564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } }
I20260807 03:13:07.706249 2565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } } }
I20260807 03:13:07.706692 2565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:07.707306 2566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a53acbae1f24a0e97f0a0e4a7b884f5. Latest consensus state: current_term: 3 leader_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a53acbae1f24a0e97f0a0e4a7b884f5" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 40745 } } }
I20260807 03:13:07.707630 2566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5 [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:07.709908 2571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:13:07.720487 2571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:13:07.722424 2571 catalog_manager.cc:1259] Loaded cluster ID: b7b3db2bd55248b39d55a906a1aadc35
I20260807 03:13:07.722592 2571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:13:07.727843 2571 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:13:07.729947 2571 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Loaded TSK: 0
I20260807 03:13:07.730149 2571 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4a53acbae1f24a0e97f0a0e4a7b884f5: Loaded TSK: 1
F20260807 03:13:07.731561 2571 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f700afc99ad google::LogMessage::Fail() at ??:0
@ 0x7f700afcc88a google::LogMessage::SendToLog() at ??:0
@ 0x7f700afc94de google::LogMessage::Flush() at ??:0
@ 0x7f700afcaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f700ed2d7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f700ed1c73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f70209f81c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f70209f7783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7020a832a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f7020a830c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f7020acd5e9 std::function<>::operator()() at ??:0
@ 0x7f70209fbcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f70209f9c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7020a81610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f7020a81439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f700dc15345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f700dc1891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f700dc18799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f700dbf3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7010ecd6db start_thread at ??:0
@ 0x7f7009e6361f clone at ??:0
W20260807 03:13:08.013041 2562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:08.013367 2562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:08.013545 2562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:08.020954 2562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:08.021103 2562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:08.021278 2562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:13:08.031028 2562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:42549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=36277
--tserver_master_addrs=127.17.31.62:40745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:08.032308 2562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:08.033665 2562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:08.046963 2608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:08.046958 2611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:08.047097 2609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:08.048138 2562 server_base.cc:1034] running on GCE node
I20260807 03:13:08.048663 2562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:08.049479 2562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:08.050669 2562 hybrid_clock.cc:648] HybridClock initialized: now 1786072388050620 us; error 48 us; skew 500 ppm
I20260807 03:13:08.050974 2562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:08 dist-test-slave-tm1g krb5kdc[1366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072388, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:08.054121 2562 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:13:08.055585 2562 webserver.cc:460] Webserver started at http://127.17.31.1:36277/ using document root <none> and password file <none>
I20260807 03:13:08.056243 2562 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:08.056394 2562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:08.061492 2562 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260807 03:13:08.064415 2618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:08.065750 2562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:13:08.065948 2562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "d6a772b69d794e7b87f94cc0af808b20"
format_stamp: "Formatted at 2026-08-07 03:12:22 on dist-test-slave-tm1g"
server_key: "bea1f651ca29b029ac52866a3bccfe5f"
server_key_iv: "af02a5e05b7a2700e11327014b2b0573"
server_key_version: "encryptionkey@0"
I20260807 03:13:08.067111 2562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:08.086737 2562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:08.090487 2562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:08.090783 2562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:08.091775 2562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:08.093489 2625 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 03:13:08.099071 2562 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 03:13:08.099231 2562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260807 03:13:08.099371 2562 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 03:13:08.102452 2562 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 03:13:08.102552 2562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260807 03:13:08.102835 2625 tablet_bootstrap.cc:492] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20: Bootstrap starting.
I20260807 03:13:08.129865 2625 log.cc:824] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:08.142990 2562 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:42549
I20260807 03:13:08.143029 2735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:42549 every 8 connection(s)
I20260807 03:13:08.144325 2562 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:13:08.144825 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 2562
I20260807 03:13:08.145823 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:34983
--local_ip_for_outbound_sockets=127.17.31.2
--tserver_master_addrs=127.17.31.62:40745
--webserver_port=41107
--webserver_interface=127.17.31.2
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:13:08.148886 2625 tablet_bootstrap.cc:492] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 03:13:08.149559 2625 tablet_bootstrap.cc:492] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20: Bootstrap complete.
I20260807 03:13:08.149683 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:08.147193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:48717)
0807 03:13:08.148090 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:08.148379 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":414,"threads_started":1}
W20260807 03:13:08.150333 2736 heartbeater.cc:644] Failed to heartbeat to 127.17.31.62:40745 (0 consecutive failures): Network error: Failed to ping master at 127.17.31.62:40745: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
I20260807 03:13:08.151214 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:08.150817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:46237)
0807 03:13:08.150969 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:08.151055 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 03:13:08.152375 2625 ts_tablet_manager.cc:1397] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20: Time spent bootstrapping tablet: real 0.050s user 0.038s sys 0.008s
I20260807 03:13:08.154878 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:08.151788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:49393)
0807 03:13:08.154552 (+ 2764us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:08.154671 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2651,"threads_started":1}
W20260807 03:13:08.158836 2736 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 03:13:08.161983 2625 raft_consensus.cc:348] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } } peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } }
I20260807 03:13:08.164237 2625 raft_consensus.cc:729] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a772b69d794e7b87f94cc0af808b20, State: Initialized, Role: FOLLOWER
I20260807 03:13:08.164795 2625 consensus_queue.cc:260] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } } peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } }
I20260807 03:13:08.166391 2625 ts_tablet_manager.cc:1428] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260807 03:13:08.166770 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:08.166119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:48275)
0807 03:13:08.166351 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:08.166581 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260807 03:13:08.167661 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:08.167357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:59695)
0807 03:13:08.167501 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:08.167570 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 03:13:08.485517 2740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:08.485836 2740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:08.485991 2740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:08.493803 2740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:08.493937 2740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:08.494122 2740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:13:08.503815 2740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:34983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=41107
--tserver_master_addrs=127.17.31.62:40745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:08.505120 2740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:08.506523 2740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:08.519693 2747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:08.519598 2748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:08.519654 2750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:08.520790 2740 server_base.cc:1034] running on GCE node
I20260807 03:13:08.521343 2740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:08.522215 2740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:08.523437 2740 hybrid_clock.cc:648] HybridClock initialized: now 1786072388523413 us; error 72 us; skew 500 ppm
I20260807 03:13:08.523746 2740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:08 dist-test-slave-tm1g krb5kdc[1366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072388, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:08.526813 2740 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:13:08.528159 2740 webserver.cc:460] Webserver started at http://127.17.31.2:41107/ using document root <none> and password file <none>
I20260807 03:13:08.528851 2740 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:08.529002 2740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:08.534147 2740 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:13:08.536733 2757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:08.538023 2740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:13:08.538198 2740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "059b6926da8c42f6bc96c7e97a141363"
format_stamp: "Formatted at 2026-08-07 03:12:22 on dist-test-slave-tm1g"
server_key: "6e36f5fc9379f5b40cf60550ff93416a"
server_key_iv: "9d2804ca423099098de6e6cd9803c751"
server_key_version: "encryptionkey@0"
I20260807 03:13:08.539232 2740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:08.567658 2740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:08.571857 2740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:08.572139 2740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:08.573086 2740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:08.574389 2740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:08.574519 2740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:08.574653 2740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:08.574743 2740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:08.608264 2740 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:34983
I20260807 03:13:08.608353 2874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:34983 every 8 connection(s)
I20260807 03:13:08.609730 2740 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:13:08.610435 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 2740
I20260807 03:13:08.611248 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:36549
--local_ip_for_outbound_sockets=127.17.31.3
--tserver_master_addrs=127.17.31.62:40745
--webserver_port=38191
--webserver_interface=127.17.31.3
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:13:08.614987 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:08.612853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:34125)
0807 03:13:08.613702 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:08.614000 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":368,"threads_started":1}
W20260807 03:13:08.615713 2875 heartbeater.cc:644] Failed to heartbeat to 127.17.31.62:40745 (0 consecutive failures): Network error: Failed to ping master at 127.17.31.62:40745: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
I20260807 03:13:08.616597 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:08.616285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:55635)
0807 03:13:08.616426 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:08.616493 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 03:13:08.617389 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:08.617078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:44473)
0807 03:13:08.617212 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:08.617291 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260807 03:13:08.617590 2875 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 03:13:08.951383 2878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:08.951668 2878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:08.951825 2878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:08.959126 2878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:08.959229 2878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:08.959430 2878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:13:08.969120 2878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:36549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=38191
--tserver_master_addrs=127.17.31.62:40745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:08.970498 2878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:08.971845 2878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:08.985165 2884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:08.985210 2886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:08.985210 2883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:08.986052 2878 server_base.cc:1034] running on GCE node
I20260807 03:13:08.986640 2878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:08.987468 2878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:08.988662 2878 hybrid_clock.cc:648] HybridClock initialized: now 1786072388988620 us; error 34 us; skew 500 ppm
I20260807 03:13:08.988965 2878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:08 dist-test-slave-tm1g krb5kdc[1366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072388, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:08.991984 2878 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:13:08.993402 2878 webserver.cc:460] Webserver started at http://127.17.31.3:38191/ using document root <none> and password file <none>
I20260807 03:13:08.994060 2878 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:08.994221 2878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:08.999611 2878 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260807 03:13:09.002197 2893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:09.003477 2878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:13:09.003654 2878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "65a813aa28bc4313883d12f358cdd313"
format_stamp: "Formatted at 2026-08-07 03:12:23 on dist-test-slave-tm1g"
server_key: "1d4f9c6a3bd0742703b2014955c7d015"
server_key_iv: "98534258f4ee3cfdedbf00333722a11c"
server_key_version: "encryptionkey@0"
I20260807 03:13:09.004611 2878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:09.020752 2878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:09.024374 2878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:09.024647 2878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:09.025573 2878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:09.026860 2878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:09.026980 2878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:09.027112 2878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:09.027204 2878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:09.061293 2878 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:36549
I20260807 03:13:09.061388 3009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:36549 every 8 connection(s)
I20260807 03:13:09.062803 2878 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:13:09.067512 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:09.065660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:34207)
0807 03:13:09.066447 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:09.066711 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":274,"threads_started":1}
W20260807 03:13:09.068203 3010 heartbeater.cc:644] Failed to heartbeat to 127.17.31.62:40745 (0 consecutive failures): Network error: Failed to ping master at 127.17.31.62:40745: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
I20260807 03:13:09.068969 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:09.068634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:51239)
0807 03:13:09.068778 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:09.068859 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 03:13:09.069370 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 2878
I20260807 03:13:09.069841 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:09.069515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:47867)
0807 03:13:09.069653 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:09.069731 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 03:13:09.070523 3010 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 03:13:09.169303 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:09.168453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:44653)
0807 03:13:09.168877 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:09.169016 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":219,"threads_started":1}
I20260807 03:13:09.519619 3015 raft_consensus.cc:482] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:13:09.519904 3015 raft_consensus.cc:504] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } } peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } }
I20260807 03:13:09.521490 3015 leader_election.cc:288] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 059b6926da8c42f6bc96c7e97a141363 (127.17.31.2:34983), 65a813aa28bc4313883d12f358cdd313 (127.17.31.3:36549)
Aug 07 03:13:09 dist-test-slave-tm1g krb5kdc[1366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072388, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
Aug 07 03:13:09 dist-test-slave-tm1g krb5kdc[1366](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072388, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:13:09.542966 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:09.522286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:42227 (local address 127.17.31.3:36549)
0807 03:13:09.523640 (+ 1354us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:09.523656 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:09.524200 (+ 544us) server_negotiation.cc:407] Connection header received
0807 03:13:09.525766 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:09.525774 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:09.526102 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:09.526431 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:09.528092 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:09.529477 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:09.531307 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:09.531915 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:09.532160 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:09.535891 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:09.535903 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:09.535920 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:09.535973 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:09.538838 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:09.539771 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:09.539779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:09.539786 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:09.540003 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:09.540526 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:09.540532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:09.540535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:09.541185 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:09.541390 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:09.542139 (+ 749us) server_negotiation.cc:299] Negotiation successful
0807 03:13:09.542558 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1171,"thread_start_us":1027,"threads_started":1}
I20260807 03:13:09.542984 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:09.521788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:34983 (local address 127.17.31.1:59397)
0807 03:13:09.522134 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:09.522182 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:09.522800 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:09.526377 (+ 3577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:09.526391 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:09.527219 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:09.527897 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:09.527910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:09.529726 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:09.529737 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:09.531148 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:09.531160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:09.532162 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:09.532173 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:09.532442 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:09.533187 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:13:09.533211 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:09.535976 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:09.539920 (+ 3944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:09.539928 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:09.539931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:09.540462 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:09.540901 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:09.540906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:09.540909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:09.541074 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:09.541755 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:09.541760 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:09.541926 (+ 166us) client_negotiation.cc:769] Sending connection context
0807 03:13:09.542063 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 03:13:09.542593 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":347,"spinlock_wait_cycles":20608}
I20260807 03:13:09.543040 3016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:09.521971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:36549 (local address 127.17.31.1:42227)
0807 03:13:09.524079 (+ 2108us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:09.524096 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:09.524221 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:09.526394 (+ 2173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:09.526399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:09.527206 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:09.527891 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:09.527909 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:09.529656 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:09.529660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:09.531155 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:09.531168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:09.532200 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:09.532208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:09.532444 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:09.533251 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:13:09.533268 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:09.535573 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:09.538982 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:09.538993 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:09.539007 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:09.539617 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:09.540129 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:09.540133 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:09.540137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:09.540370 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:09.541338 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:09.541345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:09.541648 (+ 303us) client_negotiation.cc:769] Sending connection context
0807 03:13:09.541900 (+ 252us) client_negotiation.cc:240] Negotiation successful
0807 03:13:09.542436 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1985,"mutex_wait_us":166,"spinlock_wait_cycles":1920,"thread_start_us":1851,"threads_started":1}
I20260807 03:13:09.543344 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:09.521992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:59397 (local address 127.17.31.2:34983)
0807 03:13:09.523952 (+ 1960us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:09.523966 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:09.524008 (+ 42us) server_negotiation.cc:407] Connection header received
0807 03:13:09.525560 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:09.525568 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:09.525916 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:09.526241 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:09.528082 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:09.529558 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:09.531285 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:09.531913 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:09.532120 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:09.536224 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:09.536231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:09.536241 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:09.536274 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:09.539791 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:09.540615 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:09.540621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:09.540626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:09.540779 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:09.541188 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:09.541191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:09.541193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:09.541623 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:09.541827 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:09.542506 (+ 679us) server_negotiation.cc:299] Negotiation successful
0807 03:13:09.542910 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1786,"thread_start_us":1644,"threads_started":1}
W20260807 03:13:09.546248 2619 leader_election.cc:341] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 65a813aa28bc4313883d12f358cdd313 (127.17.31.3:36549): Not found: Tablet not found: 8cdcf3c52ec54cd78de6138f5298619b
W20260807 03:13:09.546526 2620 leader_election.cc:341] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 059b6926da8c42f6bc96c7e97a141363 (127.17.31.2:34983): Not found: Tablet not found: 8cdcf3c52ec54cd78de6138f5298619b
I20260807 03:13:09.546703 2620 leader_election.cc:302] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6a772b69d794e7b87f94cc0af808b20; no voters: 059b6926da8c42f6bc96c7e97a141363, 65a813aa28bc4313883d12f358cdd313
I20260807 03:13:09.547127 3015 raft_consensus.cc:2726] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:13:09.618901 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:09.618047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:40905)
0807 03:13:09.618525 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:09.618667 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":197,"threads_started":1}
I20260807 03:13:10.071856 3020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:10.071066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:47303)
0807 03:13:10.071522 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:10.071664 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":195,"threads_started":1}
I20260807 03:13:10.170912 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:10.170163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:39381)
0807 03:13:10.170618 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:10.170698 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":258,"threads_started":1}
I20260807 03:13:10.620358 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:10.619707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:42897)
0807 03:13:10.620120 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:10.620210 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":217,"threads_started":1}
I20260807 03:13:11.073320 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:11.072661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:37013)
0807 03:13:11.073067 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:11.073149 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":222,"threads_started":1}
I20260807 03:13:11.172506 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:11.171760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:58655)
0807 03:13:11.172248 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:11.172326 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":227,"threads_started":1}
I20260807 03:13:11.535121 3025 raft_consensus.cc:482] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:13:11.535353 3025 raft_consensus.cc:504] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } } peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } }
I20260807 03:13:11.536058 3025 leader_election.cc:288] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 059b6926da8c42f6bc96c7e97a141363 (127.17.31.2:34983), 65a813aa28bc4313883d12f358cdd313 (127.17.31.3:36549)
W20260807 03:13:11.537107 2620 leader_election.cc:341] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 059b6926da8c42f6bc96c7e97a141363 (127.17.31.2:34983): Not found: Tablet not found: 8cdcf3c52ec54cd78de6138f5298619b
W20260807 03:13:11.537300 2619 leader_election.cc:341] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 65a813aa28bc4313883d12f358cdd313 (127.17.31.3:36549): Not found: Tablet not found: 8cdcf3c52ec54cd78de6138f5298619b
I20260807 03:13:11.537490 2619 leader_election.cc:302] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6a772b69d794e7b87f94cc0af808b20; no voters: 059b6926da8c42f6bc96c7e97a141363, 65a813aa28bc4313883d12f358cdd313
I20260807 03:13:11.537752 3025 raft_consensus.cc:2726] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:13:11.621989 3026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:11.621323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:55629)
0807 03:13:11.621756 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:11.621830 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":240,"threads_started":1}
I20260807 03:13:12.074926 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:12.074164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:43145)
0807 03:13:12.074669 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:12.074757 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":283,"threads_started":1}
I20260807 03:13:12.173964 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:12.173314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:60085)
0807 03:13:12.173705 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:12.173794 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":204,"threads_started":1}
I20260807 03:13:12.623553 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:12.622876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:42113)
0807 03:13:12.623299 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:12.623372 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":216,"threads_started":1}
I20260807 03:13:13.076467 3030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:13.075741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:44881)
0807 03:13:13.076181 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:13.076260 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":206,"threads_started":1}
I20260807 03:13:13.175674 3031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:13.174980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:35377)
0807 03:13:13.175419 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:13.175494 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":245,"threads_started":1}
I20260807 03:13:13.277323 3032 raft_consensus.cc:482] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:13:13.277597 3032 raft_consensus.cc:504] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } } peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } }
I20260807 03:13:13.278436 3032 leader_election.cc:288] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 059b6926da8c42f6bc96c7e97a141363 (127.17.31.2:34983), 65a813aa28bc4313883d12f358cdd313 (127.17.31.3:36549)
W20260807 03:13:13.279187 2619 leader_election.cc:341] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 65a813aa28bc4313883d12f358cdd313 (127.17.31.3:36549): Not found: Tablet not found: 8cdcf3c52ec54cd78de6138f5298619b
W20260807 03:13:13.279333 2620 leader_election.cc:341] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 059b6926da8c42f6bc96c7e97a141363 (127.17.31.2:34983): Not found: Tablet not found: 8cdcf3c52ec54cd78de6138f5298619b
I20260807 03:13:13.279505 2620 leader_election.cc:302] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6a772b69d794e7b87f94cc0af808b20; no voters: 059b6926da8c42f6bc96c7e97a141363, 65a813aa28bc4313883d12f358cdd313
I20260807 03:13:13.279754 3032 raft_consensus.cc:2726] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:13:13.625120 3033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:13.624463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:52391)
0807 03:13:13.624892 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:13.624970 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":221,"threads_started":1}
I20260807 03:13:14.077976 3034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:14.077308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:60359)
0807 03:13:14.077749 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:14.077829 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":257,"threads_started":1}
I20260807 03:13:14.177227 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:14.176567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:41465)
0807 03:13:14.176990 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:14.177075 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":232,"threads_started":1}
I20260807 03:13:14.626804 3036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:14.626051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:55137)
0807 03:13:14.626545 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:14.626621 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":286,"threads_started":1}
I20260807 03:13:15.079536 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:15.078827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:40991)
0807 03:13:15.079262 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:15.079348 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":242,"threads_started":1}
I20260807 03:13:15.178993 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:15.178113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:47391)
0807 03:13:15.178722 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:15.178800 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":351,"threads_started":1}
I20260807 03:13:15.628530 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:15.627719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:58323)
0807 03:13:15.628259 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:15.628334 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":240,"threads_started":1}
I20260807 03:13:16.081287 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:16.080523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:52607)
0807 03:13:16.080992 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:16.081093 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":233,"threads_started":1}
I20260807 03:13:16.180464 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:16.179805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:39665)
0807 03:13:16.180222 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:16.180310 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":222,"threads_started":1}
I20260807 03:13:16.630089 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:16.629354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:48751)
0807 03:13:16.629772 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:16.629849 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":212,"threads_started":1}
I20260807 03:13:17.083153 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:17.082376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:33897)
0807 03:13:17.082883 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:17.082962 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":341,"threads_started":1}
I20260807 03:13:17.182123 3045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:17.181328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:53749)
0807 03:13:17.181841 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:17.181933 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":284,"threads_started":1}
I20260807 03:13:17.631978 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:17.631083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:38895)
0807 03:13:17.631649 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:17.631766 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":283,"threads_started":1}
I20260807 03:13:18.084753 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:18.084050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:50301)
0807 03:13:18.084513 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:18.084597 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":239,"threads_started":1}
I20260807 03:13:18.183795 3048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:18.183099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:58781)
0807 03:13:18.183557 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:18.183637 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":245,"threads_started":1}
I20260807 03:13:18.334918 3049 raft_consensus.cc:482] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:13:18.335150 3049 raft_consensus.cc:504] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a772b69d794e7b87f94cc0af808b20" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42549 } } peers { permanent_uuid: "059b6926da8c42f6bc96c7e97a141363" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 34983 } } peers { permanent_uuid: "65a813aa28bc4313883d12f358cdd313" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36549 } }
I20260807 03:13:18.335832 3049 leader_election.cc:288] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 059b6926da8c42f6bc96c7e97a141363 (127.17.31.2:34983), 65a813aa28bc4313883d12f358cdd313 (127.17.31.3:36549)
W20260807 03:13:18.336728 2619 leader_election.cc:341] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 65a813aa28bc4313883d12f358cdd313 (127.17.31.3:36549): Not found: Tablet not found: 8cdcf3c52ec54cd78de6138f5298619b
W20260807 03:13:18.336894 2620 leader_election.cc:341] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 059b6926da8c42f6bc96c7e97a141363 (127.17.31.2:34983): Not found: Tablet not found: 8cdcf3c52ec54cd78de6138f5298619b
I20260807 03:13:18.337011 2620 leader_election.cc:302] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6a772b69d794e7b87f94cc0af808b20; no voters: 059b6926da8c42f6bc96c7e97a141363, 65a813aa28bc4313883d12f358cdd313
I20260807 03:13:18.337231 3049 raft_consensus.cc:2726] T 8cdcf3c52ec54cd78de6138f5298619b P d6a772b69d794e7b87f94cc0af808b20 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 03:13:18.633571 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:18.632917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:39751)
0807 03:13:18.633323 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:18.633398 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":218,"threads_started":1}
I20260807 03:13:19.086287 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:19.085641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:60773)
0807 03:13:19.086043 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:19.086119 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":221,"threads_started":1}
I20260807 03:13:19.185436 3052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:19.184663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:57153)
0807 03:13:19.185128 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:19.185216 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":246,"threads_started":1}
I20260807 03:13:19.635390 3053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:19.634536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:37899)
0807 03:13:19.635020 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:19.635159 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":238,"threads_started":1}
I20260807 03:13:20.087864 3054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:20.087159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:36979)
0807 03:13:20.087630 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:20.087708 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":236,"threads_started":1}
I20260807 03:13:20.186964 3055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:20.186234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:38035)
0807 03:13:20.186725 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:20.186804 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":274,"threads_started":1}
I20260807 03:13:20.637075 3056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:20.636366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:55665)
0807 03:13:20.636831 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:20.636908 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":249,"threads_started":1}
I20260807 03:13:21.089458 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:21.088749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:34391)
0807 03:13:21.089188 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:21.089266 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":237,"threads_started":1}
I20260807 03:13:21.188485 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:21.187792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:36075)
0807 03:13:21.188219 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:21.188295 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":233,"threads_started":1}
I20260807 03:13:21.638878 3059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:21.638050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:56903)
0807 03:13:21.638627 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:21.638716 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":347,"threads_started":1}
I20260807 03:13:22.091148 3060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:22.090490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:33263)
0807 03:13:22.090906 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:22.090989 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":212,"threads_started":1}
I20260807 03:13:22.190039 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:22.189361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:45255)
0807 03:13:22.189783 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:22.189865 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":217,"threads_started":1}
I20260807 03:13:22.640674 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:22.639798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:60483)
0807 03:13:22.640382 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:22.640465 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":254,"threads_started":1}
I20260807 03:13:23.092723 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:23.091960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:32835)
0807 03:13:23.092447 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:23.092525 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":250,"threads_started":1}
I20260807 03:13:23.191958 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:23.191160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.1:33061)
0807 03:13:23.191596 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:23.191734 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":223,"threads_started":1}
I20260807 03:13:23.642251 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:23.641610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.2:50377)
0807 03:13:23.642009 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:23.642103 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":199,"threads_started":1}
I20260807 03:13:24.072070 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 2562
I20260807 03:13:24.091778 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:24.072982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34238 (local address 127.17.31.1:42549)
0807 03:13:24.074083 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:24.074095 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:24.074127 (+ 32us) server_negotiation.cc:407] Connection header received
0807 03:13:24.074218 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:24.074230 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:24.074569 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:24.074769 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:24.076238 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:24.077200 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:24.078513 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:24.079096 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:24.079245 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:24.086083 (+ 6838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:24.086094 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:24.086107 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:24.086138 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:24.088900 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:24.089694 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:24.089701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:24.089703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:24.089833 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:24.090194 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:24.090199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:24.090203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:24.090578 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:24.090916 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:24.091197 (+ 281us) server_negotiation.cc:299] Negotiation successful
0807 03:13:24.091472 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":238,"threads_started":1}
I20260807 03:13:24.092923 2731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:24.094861 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:24.093847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:40745 (local address 127.17.31.3:52999)
0807 03:13:24.094397 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:24.094572 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.62:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":242,"threads_started":1}
I20260807 03:13:24.181460 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 2562
I20260807 03:13:24.200812 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 2740
I20260807 03:13:24.213696 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:24.201545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48046 (local address 127.17.31.2:34983)
0807 03:13:24.202068 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:24.202072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:24.202087 (+ 15us) server_negotiation.cc:407] Connection header received
0807 03:13:24.202138 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:24.202141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:24.202196 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:24.202277 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:24.203597 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:24.204390 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:24.205583 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:24.205996 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:24.206094 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:24.208656 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:24.208661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:24.208664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:24.208693 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:24.211248 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:24.211928 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:24.211932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:24.211935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:24.212066 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:24.212416 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:24.212419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:24.212421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:24.212658 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:24.212853 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:24.213103 (+ 250us) server_negotiation.cc:299] Negotiation successful
0807 03:13:24.213259 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":287,"threads_started":1}
I20260807 03:13:24.214653 2870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:24.296015 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 2740
I20260807 03:13:24.314600 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 2878
I20260807 03:13:24.327852 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:24.315421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.17.31.3:36549)
0807 03:13:24.316053 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:24.316058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:24.316075 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:13:24.316132 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:24.316135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:24.316201 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:24.316295 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:24.317682 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:24.318587 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:24.319780 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:24.320210 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:24.320300 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:24.322946 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:24.322951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:24.322954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:24.322981 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:24.325511 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:24.326116 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:24.326120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:24.326122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:24.326255 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:24.326629 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:24.326633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:24.326634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:24.326868 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:24.327081 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:24.327307 (+ 226us) server_negotiation.cc:299] Negotiation successful
0807 03:13:24.327455 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":364,"threads_started":1}
I20260807 03:13:24.328545 3005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:24.412178 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 2878
2026-08-07T03:13:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64437 ms)
[----------] 30 tests from SecurityITest (297356 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:13:24 dist-test-slave-tm1g krb5kdc[3075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:13:24 dist-test-slave-tm1g krb5kdc[3075](info): set up 2 sockets
Aug 07 03:13:24 dist-test-slave-tm1g krb5kdc[3075](info): commencing operation
krb5kdc: starting...
W20260807 03:13:25.539986 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:13:25 dist-test-slave-tm1g krb5kdc[3075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:13:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:13:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:13:25.782104 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:46799
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:46799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:26.100500 3089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:26.100769 3089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:26.100857 3089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:26.107976 3089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:13:26.108089 3089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:26.108134 3089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:26.108184 3089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:13:26.108220 3089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:13:26.117245 3089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:46799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:46799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:26.118510 3089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:26.119735 3089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:26.131855 3094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:26.131821 3095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:26.132102 3089 server_base.cc:1034] running on GCE node
W20260807 03:13:26.132146 3097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:26.133181 3089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:26.134536 3089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:26.135757 3089 hybrid_clock.cc:648] HybridClock initialized: now 1786072406135714 us; error 37 us; skew 500 ppm
I20260807 03:13:26.136094 3089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:26 dist-test-slave-tm1g krb5kdc[3075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072406, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:26.139331 3089 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:13:26.140694 3089 webserver.cc:460] Webserver started at http://127.17.31.62:33267/ using document root <none> and password file <none>
I20260807 03:13:26.141372 3089 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:26.141539 3089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:26.141839 3089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:26.144750 3089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "56bb9be6758148d4a0fdb6cff6f0d0e0"
format_stamp: "Formatted at 2026-08-07 03:13:26 on dist-test-slave-tm1g"
server_key: "724dd9ba6e056829aa9a4c80923c9c05"
server_key_iv: "f0023d90e8c74dde64c512cae998642b"
server_key_version: "encryptionkey@0"
I20260807 03:13:26.145525 3089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "56bb9be6758148d4a0fdb6cff6f0d0e0"
format_stamp: "Formatted at 2026-08-07 03:13:26 on dist-test-slave-tm1g"
server_key: "724dd9ba6e056829aa9a4c80923c9c05"
server_key_iv: "f0023d90e8c74dde64c512cae998642b"
server_key_version: "encryptionkey@0"
I20260807 03:13:26.150429 3089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 03:13:26.153666 3104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:26.154959 3089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:13:26.155135 3089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "56bb9be6758148d4a0fdb6cff6f0d0e0"
format_stamp: "Formatted at 2026-08-07 03:13:26 on dist-test-slave-tm1g"
server_key: "724dd9ba6e056829aa9a4c80923c9c05"
server_key_iv: "f0023d90e8c74dde64c512cae998642b"
server_key_version: "encryptionkey@0"
I20260807 03:13:26.155320 3089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:26.168242 3089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:26.172290 3089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:26.172564 3089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:26.190521 3089 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:46799
I20260807 03:13:26.190553 3156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:46799 every 8 connection(s)
I20260807 03:13:26.191771 3089 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:13:26.195088 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 3089
I20260807 03:13:26.195304 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:13:26.195330 3157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:26.195966 17532 external_mini_cluster.cc:1428] Setting key 5867f390442f420380b066aab816b62f
I20260807 03:13:26.208743 3157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0: Bootstrap starting.
Aug 07 03:13:26 dist-test-slave-tm1g krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:13:26.211834 3157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:26.213001 3157 log.cc:824] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:26.216253 3157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0: No bootstrap required, opened a new log
I20260807 03:13:26.220749 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:26.197860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59858 (local address 127.17.31.62:46799)
0807 03:13:26.198690 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:26.198705 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:26.198755 (+ 50us) server_negotiation.cc:407] Connection header received
0807 03:13:26.201481 (+ 2726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:26.201514 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:26.201954 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:26.202378 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:26.203678 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:26.205684 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:26.207024 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:26.208005 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:26.208183 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:26.212684 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:26.212691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:26.212703 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:26.212741 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:26.216406 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:26.217177 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:26.217185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:26.217192 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:26.217395 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:26.217771 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:26.217778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:26.217785 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:26.218470 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:26.218794 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:26.219099 (+ 305us) server_negotiation.cc:299] Negotiation successful
0807 03:13:26.219518 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":331,"threads_started":1}
I20260807 03:13:26.224910 3157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bb9be6758148d4a0fdb6cff6f0d0e0" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46799 } }
I20260807 03:13:26.225247 3157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:26.225356 3157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56bb9be6758148d4a0fdb6cff6f0d0e0, State: Initialized, Role: FOLLOWER
I20260807 03:13:26.225914 3157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bb9be6758148d4a0fdb6cff6f0d0e0" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46799 } }
I20260807 03:13:26.226135 3157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:13:26.226289 3157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:13:26.226519 3157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:26.228955 3157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bb9be6758148d4a0fdb6cff6f0d0e0" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46799 } }
I20260807 03:13:26.229434 3157 leader_election.cc:302] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56bb9be6758148d4a0fdb6cff6f0d0e0; no voters:
I20260807 03:13:26.230082 3157 leader_election.cc:288] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:13:26.230187 3163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:26.231413 3163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [term 1 LEADER]: Becoming Leader. State: Replica: 56bb9be6758148d4a0fdb6cff6f0d0e0, State: Running, Role: LEADER
I20260807 03:13:26.231889 3163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bb9be6758148d4a0fdb6cff6f0d0e0" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46799 } }
I20260807 03:13:26.231992 3157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:13:26.234514 3165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56bb9be6758148d4a0fdb6cff6f0d0e0. Latest consensus state: current_term: 1 leader_uuid: "56bb9be6758148d4a0fdb6cff6f0d0e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bb9be6758148d4a0fdb6cff6f0d0e0" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46799 } } }
I20260807 03:13:26.234808 3164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56bb9be6758148d4a0fdb6cff6f0d0e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56bb9be6758148d4a0fdb6cff6f0d0e0" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46799 } } }
I20260807 03:13:26.235054 3165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:26.235114 3164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0 [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:26.235970 3172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:13:26.240588 3172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:13:26.248739 3172 catalog_manager.cc:1347] Generated new cluster ID: f5c9867f3d734a72ba4b9a7c5089b514
I20260807 03:13:26.248842 3172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:13:26.267082 3172 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:13:26.268771 3172 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:13:26.278781 3172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0: Generated new TSK 0
I20260807 03:13:26.279682 3172 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:13:26.346386 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:46799
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:26.669319 3184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:26.669602 3184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:26.669764 3184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:26.676509 3184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:26.676640 3184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:26.676795 3184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:13:26.685830 3184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:46799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:26.687033 3184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:26.688297 3184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:26.700834 3190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:26.700908 3189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:26.700911 3192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:26.701664 3184 server_base.cc:1034] running on GCE node
I20260807 03:13:26.702250 3184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:26.703212 3184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:26.704433 3184 hybrid_clock.cc:648] HybridClock initialized: now 1786072406704385 us; error 34 us; skew 500 ppm
I20260807 03:13:26.704813 3184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:26 dist-test-slave-tm1g krb5kdc[3075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072406, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:26.708005 3184 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:13:26.709278 3184 webserver.cc:460] Webserver started at http://127.17.31.1:42055/ using document root <none> and password file <none>
I20260807 03:13:26.709980 3184 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:26.710119 3184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:26.710508 3184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:26.713177 3184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "e7c840b99e254d648945bddf74c6cfc8"
format_stamp: "Formatted at 2026-08-07 03:13:26 on dist-test-slave-tm1g"
server_key: "e5eccb31a6633808955b87e4c804d602"
server_key_iv: "e135fb723b227c58a7bf6532f29c015e"
server_key_version: "encryptionkey@0"
I20260807 03:13:26.713907 3184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "e7c840b99e254d648945bddf74c6cfc8"
format_stamp: "Formatted at 2026-08-07 03:13:26 on dist-test-slave-tm1g"
server_key: "e5eccb31a6633808955b87e4c804d602"
server_key_iv: "e135fb723b227c58a7bf6532f29c015e"
server_key_version: "encryptionkey@0"
I20260807 03:13:26.718765 3184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260807 03:13:26.722150 3199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:26.723395 3184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:13:26.723630 3184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "e7c840b99e254d648945bddf74c6cfc8"
format_stamp: "Formatted at 2026-08-07 03:13:26 on dist-test-slave-tm1g"
server_key: "e5eccb31a6633808955b87e4c804d602"
server_key_iv: "e135fb723b227c58a7bf6532f29c015e"
server_key_version: "encryptionkey@0"
I20260807 03:13:26.723847 3184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:26.740928 3184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:26.744614 3184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:26.744896 3184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:26.745786 3184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:26.747109 3184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:26.747215 3184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:26.747363 3184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:26.747437 3184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:26.780691 3184 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:38163
I20260807 03:13:26.780723 3315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:38163 every 8 connection(s)
I20260807 03:13:26.781941 3184 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:13:26.791074 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 3184
I20260807 03:13:26.791294 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:13:26.791947 17532 external_mini_cluster.cc:1428] Setting key cfc6e11b8c491222bf71adcee22efc28
Aug 07 03:13:26 dist-test-slave-tm1g krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072406, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:13:26.801374 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:26.784427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:53447 (local address 127.17.31.62:46799)
0807 03:13:26.784878 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:26.784882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:26.785851 (+ 969us) server_negotiation.cc:407] Connection header received
0807 03:13:26.787673 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:26.787680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:26.787769 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:26.787889 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:26.789289 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:26.789977 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:26.791691 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:26.792153 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:26.792279 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:26.795902 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:26.795909 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:26.795914 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:26.795974 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:26.798496 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:26.799189 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:26.799194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:26.799197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:26.799315 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:26.799765 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:26.799770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:26.799773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:26.800020 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:26.800169 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:26.800799 (+ 630us) server_negotiation.cc:299] Negotiation successful
0807 03:13:26.801003 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":238,"threads_started":1}
I20260807 03:13:26.802235 3321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:26.784935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:46799 (local address 127.17.31.1:53447)
0807 03:13:26.785697 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:26.785742 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:26.787416 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:26.788115 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:26.788122 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:26.788606 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:26.789125 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:26.789141 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:26.790152 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:26.790162 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:26.791554 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:26.791566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:26.792362 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:26.792371 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:26.792679 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:26.793357 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:13:26.793382 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:26.795646 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:26.798633 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:26.798641 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:26.798654 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:26.799069 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:26.799422 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:26.799428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:26.799431 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:26.799617 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:26.800144 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:26.800152 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:26.800447 (+ 295us) client_negotiation.cc:769] Sending connection context
0807 03:13:26.800687 (+ 240us) client_negotiation.cc:240] Negotiation successful
0807 03:13:26.801154 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":333,"threads_started":1}
I20260807 03:13:26.803673 3316 heartbeater.cc:347] Connected to a master server at 127.17.31.62:46799
I20260807 03:13:26.803962 3316 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:26.804652 3316 heartbeater.cc:511] Master 127.17.31.62:46799 requested a full tablet report, sending...
I20260807 03:13:26.806763 3121 ts_manager.cc:194] Registered new tserver with Master: e7c840b99e254d648945bddf74c6cfc8 (127.17.31.1:38163)
I20260807 03:13:26.808168 3121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:53447
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:13:26.850281 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:46799
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:27.175918 3325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:27.176229 3325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:27.176400 3325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:27.183229 3325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:27.183362 3325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:27.183542 3325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:13:27.196651 3325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:46799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:27.197862 3325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:27.199169 3325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:27.217262 3330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:27.219652 3333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:27.219682 3331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:27.220625 3325 server_base.cc:1034] running on GCE node
I20260807 03:13:27.221349 3325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:27.222625 3325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:27.223820 3325 hybrid_clock.cc:648] HybridClock initialized: now 1786072407223767 us; error 42 us; skew 500 ppm
I20260807 03:13:27.224275 3325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:27 dist-test-slave-tm1g krb5kdc[3075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072407, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:27.228422 3325 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:13:27.229702 3325 webserver.cc:460] Webserver started at http://127.17.31.2:34333/ using document root <none> and password file <none>
I20260807 03:13:27.230425 3325 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:27.230581 3325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:27.230986 3325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:27.235247 3325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "7bf4008691014405af445b55b83433a8"
format_stamp: "Formatted at 2026-08-07 03:13:27 on dist-test-slave-tm1g"
server_key: "9eea479b047b2e6e8469e412708b527d"
server_key_iv: "efcbebf6865cdac6d18d6b9b15627c58"
server_key_version: "encryptionkey@0"
I20260807 03:13:27.236243 3325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "7bf4008691014405af445b55b83433a8"
format_stamp: "Formatted at 2026-08-07 03:13:27 on dist-test-slave-tm1g"
server_key: "9eea479b047b2e6e8469e412708b527d"
server_key_iv: "efcbebf6865cdac6d18d6b9b15627c58"
server_key_version: "encryptionkey@0"
I20260807 03:13:27.243016 3325 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260807 03:13:27.246703 3341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:27.248421 3325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:13:27.248564 3325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "7bf4008691014405af445b55b83433a8"
format_stamp: "Formatted at 2026-08-07 03:13:27 on dist-test-slave-tm1g"
server_key: "9eea479b047b2e6e8469e412708b527d"
server_key_iv: "efcbebf6865cdac6d18d6b9b15627c58"
server_key_version: "encryptionkey@0"
I20260807 03:13:27.248760 3325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:27.271634 3325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:27.275676 3325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:27.275945 3325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:27.276865 3325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:27.278157 3325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:27.278259 3325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:27.278427 3325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:27.278501 3325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:27.311646 3325 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:41613
I20260807 03:13:27.311689 3461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:41613 every 8 connection(s)
I20260807 03:13:27.312915 3325 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:13:27.322376 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 3325
I20260807 03:13:27.322561 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:13:27.322988 17532 external_mini_cluster.cc:1428] Setting key b4c06db12e510444ae43ce385aa17857
Aug 07 03:13:27 dist-test-slave-tm1g krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072407, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:13:27.333114 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:27.315320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:57171 (local address 127.17.31.62:46799)
0807 03:13:27.315750 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:27.315755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:27.316708 (+ 953us) server_negotiation.cc:407] Connection header received
0807 03:13:27.318565 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:27.318571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:27.318673 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:27.319026 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:27.320476 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.321173 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.322850 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.323306 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.323437 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:27.326873 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:27.326878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:27.326881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:27.326911 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:27.329906 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:27.330675 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:27.330680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:27.330684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:27.330819 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:27.331244 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:27.331249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:27.331254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:27.331518 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:27.331750 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:27.332455 (+ 705us) server_negotiation.cc:299] Negotiation successful
0807 03:13:27.332734 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":203,"threads_started":1}
I20260807 03:13:27.334007 3466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:27.315886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:46799 (local address 127.17.31.2:57171)
0807 03:13:27.316562 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:27.316597 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:27.318246 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:27.319251 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:27.319259 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:27.319745 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:27.320302 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:27.320326 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.321317 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.321323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:27.322700 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:27.322712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.323516 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.323523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:27.323767 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:27.324408 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:13:27.324436 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:27.326651 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:27.330050 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:27.330058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:27.330071 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:27.330549 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:27.330930 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:27.330936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:27.330939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:27.331121 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:27.331722 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:27.331730 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:27.332035 (+ 305us) client_negotiation.cc:769] Sending connection context
0807 03:13:27.332347 (+ 312us) client_negotiation.cc:240] Negotiation successful
0807 03:13:27.332866 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":276,"threads_started":1}
I20260807 03:13:27.335440 3462 heartbeater.cc:347] Connected to a master server at 127.17.31.62:46799
I20260807 03:13:27.335774 3462 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:27.336432 3462 heartbeater.cc:511] Master 127.17.31.62:46799 requested a full tablet report, sending...
I20260807 03:13:27.337852 3121 ts_manager.cc:194] Registered new tserver with Master: 7bf4008691014405af445b55b83433a8 (127.17.31.2:41613)
I20260807 03:13:27.338732 3121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:57171
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:13:27.380681 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:46799
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:27.702412 3470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:27.702682 3470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:27.702773 3470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:27.709779 3470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:27.709875 3470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:27.709981 3470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:13:27.719344 3470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:46799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:27.720531 3470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:27.721798 3470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:27.734954 3475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:27.734915 3476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:27.735116 3478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:27.736110 3470 server_base.cc:1034] running on GCE node
I20260807 03:13:27.736668 3470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:27.737556 3470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:27.738775 3470 hybrid_clock.cc:648] HybridClock initialized: now 1786072407738723 us; error 38 us; skew 500 ppm
I20260807 03:13:27.739080 3470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:27 dist-test-slave-tm1g krb5kdc[3075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072407, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:27.743494 3470 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:13:27.746861 3470 webserver.cc:460] Webserver started at http://127.17.31.3:41285/ using document root <none> and password file <none>
I20260807 03:13:27.747583 3470 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:27.747776 3470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:27.748178 3470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:27.751151 3470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "9f591de0674940649f693e1779798a8c"
format_stamp: "Formatted at 2026-08-07 03:13:27 on dist-test-slave-tm1g"
server_key: "bbab8f7bb55db7a7fb435312dc5f6cd9"
server_key_iv: "4780fa7bbffd926f9bd27fd2dbb4c865"
server_key_version: "encryptionkey@0"
I20260807 03:13:27.751900 3470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "9f591de0674940649f693e1779798a8c"
format_stamp: "Formatted at 2026-08-07 03:13:27 on dist-test-slave-tm1g"
server_key: "bbab8f7bb55db7a7fb435312dc5f6cd9"
server_key_iv: "4780fa7bbffd926f9bd27fd2dbb4c865"
server_key_version: "encryptionkey@0"
I20260807 03:13:27.757153 3470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:13:27.760653 3485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:27.761866 3470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 03:13:27.762027 3470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "9f591de0674940649f693e1779798a8c"
format_stamp: "Formatted at 2026-08-07 03:13:27 on dist-test-slave-tm1g"
server_key: "bbab8f7bb55db7a7fb435312dc5f6cd9"
server_key_iv: "4780fa7bbffd926f9bd27fd2dbb4c865"
server_key_version: "encryptionkey@0"
I20260807 03:13:27.762199 3470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:27.781461 3470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:27.785485 3470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:27.785765 3470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:27.786751 3470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:27.788014 3470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:27.788086 3470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:27.788261 3470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:27.788314 3470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:27.810866 3316 heartbeater.cc:503] Master 127.17.31.62:46799 was elected leader, sending a full tablet report...
I20260807 03:13:27.823483 3470 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:45089
I20260807 03:13:27.823536 3604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:45089 every 8 connection(s)
I20260807 03:13:27.824762 3470 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:13:27.833639 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 3470
I20260807 03:13:27.833835 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:13:27.834456 17532 external_mini_cluster.cc:1428] Setting key 9181a5519f779d8dd1697938f67546f3
Aug 07 03:13:27 dist-test-slave-tm1g krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072407, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:13:27.844713 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:27.827225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:35953 (local address 127.17.31.62:46799)
0807 03:13:27.827400 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:27.827405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:27.828676 (+ 1271us) server_negotiation.cc:407] Connection header received
0807 03:13:27.830421 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:27.830425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:27.830502 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:27.830584 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:27.831995 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.832859 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.835163 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.835717 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.835901 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:27.839501 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:27.839506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:27.839509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:27.839540 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:27.842140 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:27.842864 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:27.842868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:27.842870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:27.842984 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:27.843336 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:27.843339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:27.843341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:27.843592 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:27.843701 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:27.844288 (+ 587us) server_negotiation.cc:299] Negotiation successful
0807 03:13:27.844451 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:13:27.845573 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:27.827789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:46799 (local address 127.17.31.3:35953)
0807 03:13:27.828495 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:27.828536 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:27.830149 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:27.830778 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:27.830785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:27.831269 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:27.831848 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:27.831866 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.832982 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.832986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:27.835006 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:27.835024 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.836002 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.836007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:27.836189 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:27.836923 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:13:27.836956 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:27.839322 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:27.842246 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:27.842251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:27.842261 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:27.842754 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:27.843070 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:27.843073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:27.843075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:27.843224 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:27.843690 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:27.843695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:27.843970 (+ 275us) client_negotiation.cc:769] Sending connection context
0807 03:13:27.844170 (+ 200us) client_negotiation.cc:240] Negotiation successful
0807 03:13:27.844529 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":265,"threads_started":1}
I20260807 03:13:27.846819 3605 heartbeater.cc:347] Connected to a master server at 127.17.31.62:46799
I20260807 03:13:27.847107 3605 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:27.847770 3605 heartbeater.cc:511] Master 127.17.31.62:46799 requested a full tablet report, sending...
I20260807 03:13:27.849325 3121 ts_manager.cc:194] Registered new tserver with Master: 9f591de0674940649f693e1779798a8c (127.17.31.3:45089)
I20260807 03:13:27.850059 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:13:27.850442 3121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:35953
I20260807 03:13:27.868257 3465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:27.855433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59860 (local address 127.17.31.62:46799)
0807 03:13:27.855616 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:27.855624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:27.856015 (+ 391us) server_negotiation.cc:407] Connection header received
0807 03:13:27.856223 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:27.856228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:27.856300 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:27.856405 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:27.857866 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.858687 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.860006 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.860409 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.860538 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:27.862973 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:27.862978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:27.862981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:27.863009 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:27.865531 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:27.866245 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:27.866250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:27.866254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:27.866441 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:27.866837 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:27.866842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:27.866846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:27.867149 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:27.867418 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:27.867716 (+ 298us) server_negotiation.cc:299] Negotiation successful
0807 03:13:27.867929 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 03:13:27.872216 3121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:13:27.879245 3121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:13:27.902926 3617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:27.890621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:41613 (local address 127.0.0.1:39488)
0807 03:13:27.891537 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:27.891584 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:27.891809 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:27.893099 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:27.893120 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:27.893151 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:27.893698 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:27.893717 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.896475 (+ 2758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.896484 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:27.899624 (+ 3140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:27.899661 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.901619 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.901625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:27.901934 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:27.902096 (+ 162us) client_negotiation.cc:769] Sending connection context
0807 03:13:27.902221 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 03:13:27.902451 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":45,"thread_start_us":1019,"threads_started":1}
I20260807 03:13:27.903366 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:27.890860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.17.31.2:41613)
0807 03:13:27.892411 (+ 1551us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:27.892422 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:27.892456 (+ 34us) server_negotiation.cc:407] Connection header received
0807 03:13:27.892567 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:27.892577 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:27.892817 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:27.893968 (+ 1151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:27.894236 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.896272 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.899888 (+ 3616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.901402 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.901546 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:27.901760 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:27.902897 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0807 03:13:27.903057 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1410,"thread_start_us":1322,"threads_started":1}
I20260807 03:13:27.907341 3619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:27.891014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:38163 (local address 127.0.0.1:60938)
0807 03:13:27.900017 (+ 9003us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:27.900034 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:27.900149 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:27.901055 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:27.901061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:27.901088 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:27.901470 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:27.901487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.903356 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.903361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:27.905127 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:27.905142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.906430 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.906437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:27.906636 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:27.906698 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 03:13:27.906791 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 03:13:27.906935 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8875,"mutex_wait_us":4,"thread_start_us":8732,"threads_started":1}
I20260807 03:13:27.907326 3393 tablet_service.cc:1467] Processing CreateTablet for tablet 44967480e6804834b261561f8b68fbb6 (DEFAULT_TABLE table=test-table [id=d2c5c152a1fc4ca4b996b51812f48f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:27.908073 3620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:27.891656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60938 (local address 127.17.31.1:38163)
0807 03:13:27.899100 (+ 7444us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:27.899109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:27.900279 (+ 1170us) server_negotiation.cc:407] Connection header received
0807 03:13:27.900395 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:27.900402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:27.900611 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:27.900778 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:27.901688 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.903220 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.905286 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.906189 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.906368 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:27.906597 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:27.907547 (+ 950us) server_negotiation.cc:299] Negotiation successful
0807 03:13:27.907715 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1034,"thread_start_us":868,"threads_started":1}
I20260807 03:13:27.908864 3393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44967480e6804834b261561f8b68fbb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:27.909262 3621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:27.892965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:45089 (local address 127.0.0.1:33610)
0807 03:13:27.900410 (+ 7445us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:27.900432 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:27.900563 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:27.901595 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:27.901604 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:27.901628 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:27.902082 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:27.902095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.904603 (+ 2508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.904608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:27.906733 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:27.906749 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.908488 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.908494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:27.908656 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:27.908686 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 03:13:27.908775 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 03:13:27.908906 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7294,"mutex_wait_us":39,"thread_start_us":7151,"threads_started":1}
I20260807 03:13:27.909307 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:27.893194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.17.31.3:45089)
0807 03:13:27.901141 (+ 7947us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:27.901146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:27.901171 (+ 25us) server_negotiation.cc:407] Connection header received
0807 03:13:27.901241 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:27.901245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:27.901387 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:27.902271 (+ 884us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:27.902561 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.904440 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.906930 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:27.908287 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:27.908430 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:27.908642 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:27.908873 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 03:13:27.909021 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7808,"thread_start_us":7655,"threads_started":1}
I20260807 03:13:27.911329 3248 tablet_service.cc:1467] Processing CreateTablet for tablet 44967480e6804834b261561f8b68fbb6 (DEFAULT_TABLE table=test-table [id=d2c5c152a1fc4ca4b996b51812f48f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:27.912400 3248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44967480e6804834b261561f8b68fbb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:27.912729 3536 tablet_service.cc:1467] Processing CreateTablet for tablet 44967480e6804834b261561f8b68fbb6 (DEFAULT_TABLE table=test-table [id=d2c5c152a1fc4ca4b996b51812f48f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:27.914054 3536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44967480e6804834b261561f8b68fbb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:27.923765 3623 tablet_bootstrap.cc:492] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8: Bootstrap starting.
I20260807 03:13:27.927070 3625 tablet_bootstrap.cc:492] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8: Bootstrap starting.
I20260807 03:13:27.928028 3623 tablet_bootstrap.cc:654] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:27.929616 3623 log.cc:824] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:27.931023 3626 tablet_bootstrap.cc:492] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c: Bootstrap starting.
I20260807 03:13:27.932118 3625 tablet_bootstrap.cc:654] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:27.932902 3623 tablet_bootstrap.cc:492] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8: No bootstrap required, opened a new log
I20260807 03:13:27.933243 3623 ts_tablet_manager.cc:1397] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260807 03:13:27.933239 3625 log.cc:824] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:27.934260 3626 tablet_bootstrap.cc:654] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:27.935541 3626 log.cc:824] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:27.941069 3626 tablet_bootstrap.cc:492] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c: No bootstrap required, opened a new log
I20260807 03:13:27.941162 3625 tablet_bootstrap.cc:492] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8: No bootstrap required, opened a new log
I20260807 03:13:27.941349 3626 ts_tablet_manager.cc:1397] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260807 03:13:27.941403 3625 ts_tablet_manager.cc:1397] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.007s
I20260807 03:13:27.943589 3623 raft_consensus.cc:348] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf4008691014405af445b55b83433a8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41613 } } peers { permanent_uuid: "e7c840b99e254d648945bddf74c6cfc8" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38163 } } peers { permanent_uuid: "9f591de0674940649f693e1779798a8c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45089 } }
I20260807 03:13:27.943948 3623 raft_consensus.cc:374] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:27.944033 3623 raft_consensus.cc:729] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf4008691014405af445b55b83433a8, State: Initialized, Role: FOLLOWER
I20260807 03:13:27.944578 3623 consensus_queue.cc:260] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf4008691014405af445b55b83433a8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41613 } } peers { permanent_uuid: "e7c840b99e254d648945bddf74c6cfc8" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38163 } } peers { permanent_uuid: "9f591de0674940649f693e1779798a8c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45089 } }
I20260807 03:13:27.945631 3462 heartbeater.cc:503] Master 127.17.31.62:46799 was elected leader, sending a full tablet report...
I20260807 03:13:27.946287 3623 ts_tablet_manager.cc:1428] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260807 03:13:27.952121 3626 raft_consensus.cc:348] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf4008691014405af445b55b83433a8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41613 } } peers { permanent_uuid: "e7c840b99e254d648945bddf74c6cfc8" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38163 } } peers { permanent_uuid: "9f591de0674940649f693e1779798a8c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45089 } }
I20260807 03:13:27.952090 3625 raft_consensus.cc:348] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf4008691014405af445b55b83433a8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41613 } } peers { permanent_uuid: "e7c840b99e254d648945bddf74c6cfc8" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38163 } } peers { permanent_uuid: "9f591de0674940649f693e1779798a8c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45089 } }
I20260807 03:13:27.952513 3625 raft_consensus.cc:374] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:27.952509 3626 raft_consensus.cc:374] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:27.952617 3626 raft_consensus.cc:729] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f591de0674940649f693e1779798a8c, State: Initialized, Role: FOLLOWER
I20260807 03:13:27.952618 3625 raft_consensus.cc:729] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c840b99e254d648945bddf74c6cfc8, State: Initialized, Role: FOLLOWER
I20260807 03:13:27.953202 3626 consensus_queue.cc:260] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf4008691014405af445b55b83433a8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41613 } } peers { permanent_uuid: "e7c840b99e254d648945bddf74c6cfc8" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38163 } } peers { permanent_uuid: "9f591de0674940649f693e1779798a8c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45089 } }
I20260807 03:13:27.953202 3625 consensus_queue.cc:260] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf4008691014405af445b55b83433a8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41613 } } peers { permanent_uuid: "e7c840b99e254d648945bddf74c6cfc8" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38163 } } peers { permanent_uuid: "9f591de0674940649f693e1779798a8c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45089 } }
I20260807 03:13:27.954217 3605 heartbeater.cc:503] Master 127.17.31.62:46799 was elected leader, sending a full tablet report...
I20260807 03:13:27.955042 3626 ts_tablet_manager.cc:1428] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260807 03:13:27.955293 3625 ts_tablet_manager.cc:1428] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
W20260807 03:13:28.034503 3317 tablet.cc:2369] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:13:28.064910 3464 tablet.cc:2369] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:13:28.076357 3606 tablet.cc:2369] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:13:28.082832 3631 raft_consensus.cc:482] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:13:28.083078 3631 raft_consensus.cc:504] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf4008691014405af445b55b83433a8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41613 } } peers { permanent_uuid: "e7c840b99e254d648945bddf74c6cfc8" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38163 } } peers { permanent_uuid: "9f591de0674940649f693e1779798a8c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45089 } }
I20260807 03:13:28.084719 3631 leader_election.cc:288] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7c840b99e254d648945bddf74c6cfc8 (127.17.31.1:38163), 9f591de0674940649f693e1779798a8c (127.17.31.3:45089)
I20260807 03:13:28.091629 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:28.085318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:33667 (local address 127.17.31.3:45089)
0807 03:13:28.085623 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:28.085631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:28.086269 (+ 638us) server_negotiation.cc:407] Connection header received
0807 03:13:28.086494 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:28.086498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:28.086555 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:28.086652 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:28.087266 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:28.087981 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:28.089809 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:28.090599 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:28.090743 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:28.090881 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:28.091224 (+ 343us) server_negotiation.cc:299] Negotiation successful
0807 03:13:28.091359 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260807 03:13:28.091639 3620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:28.085174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:60273 (local address 127.17.31.1:38163)
0807 03:13:28.085415 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:28.085423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:28.086068 (+ 645us) server_negotiation.cc:407] Connection header received
0807 03:13:28.086293 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:28.086341 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:28.086416 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:28.086514 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:28.087128 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:28.087868 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:28.089797 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:28.090628 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:28.090799 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:28.090909 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:28.091238 (+ 329us) server_negotiation.cc:299] Negotiation successful
0807 03:13:28.091381 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 03:13:28.091785 3635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:28.085215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:45089 (local address 127.17.31.2:33667)
0807 03:13:28.086160 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:28.086174 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:28.086341 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:28.086678 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:28.086683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:28.086720 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:28.087124 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:28.087133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:28.088129 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:28.088147 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:28.089659 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:28.089671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:28.090867 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:28.090873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:28.091066 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:28.091090 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 03:13:28.091197 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 03:13:28.091355 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":84,"thread_start_us":611,"threads_started":1}
I20260807 03:13:28.092125 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:28.085025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:38163 (local address 127.17.31.2:60273)
0807 03:13:28.085946 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:28.085969 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:28.086142 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:28.086531 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:28.086535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:28.086560 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:28.086972 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:28.086981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:28.087991 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:28.087999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:28.089647 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:28.089659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:28.090879 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:28.090886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:28.091080 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:28.091102 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 03:13:28.091203 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 03:13:28.091342 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":134,"spinlock_wait_cycles":16128,"thread_start_us":554,"threads_started":1}
I20260807 03:13:28.092866 3556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44967480e6804834b261561f8b68fbb6" candidate_uuid: "7bf4008691014405af445b55b83433a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f591de0674940649f693e1779798a8c" is_pre_election: true
I20260807 03:13:28.093161 3269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44967480e6804834b261561f8b68fbb6" candidate_uuid: "7bf4008691014405af445b55b83433a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c840b99e254d648945bddf74c6cfc8" is_pre_election: true
I20260807 03:13:28.093449 3556 raft_consensus.cc:2454] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf4008691014405af445b55b83433a8 in term 0.
I20260807 03:13:28.093693 3269 raft_consensus.cc:2454] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bf4008691014405af445b55b83433a8 in term 0.
I20260807 03:13:28.094185 3344 leader_election.cc:302] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bf4008691014405af445b55b83433a8, 9f591de0674940649f693e1779798a8c; no voters:
I20260807 03:13:28.094594 3631 raft_consensus.cc:2781] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:13:28.094738 3631 raft_consensus.cc:482] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:13:28.094854 3631 raft_consensus.cc:3037] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:28.096990 3631 raft_consensus.cc:504] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf4008691014405af445b55b83433a8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41613 } } peers { permanent_uuid: "e7c840b99e254d648945bddf74c6cfc8" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38163 } } peers { permanent_uuid: "9f591de0674940649f693e1779798a8c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45089 } }
I20260807 03:13:28.097646 3631 leader_election.cc:288] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [CANDIDATE]: Term 1 election: Requested vote from peers e7c840b99e254d648945bddf74c6cfc8 (127.17.31.1:38163), 9f591de0674940649f693e1779798a8c (127.17.31.3:45089)
I20260807 03:13:28.098006 3269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44967480e6804834b261561f8b68fbb6" candidate_uuid: "7bf4008691014405af445b55b83433a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c840b99e254d648945bddf74c6cfc8"
I20260807 03:13:28.098163 3556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44967480e6804834b261561f8b68fbb6" candidate_uuid: "7bf4008691014405af445b55b83433a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f591de0674940649f693e1779798a8c"
I20260807 03:13:28.098212 3269 raft_consensus.cc:3037] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:28.098400 3556 raft_consensus.cc:3037] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:28.101433 3269 raft_consensus.cc:2454] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf4008691014405af445b55b83433a8 in term 1.
I20260807 03:13:28.101508 3556 raft_consensus.cc:2454] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bf4008691014405af445b55b83433a8 in term 1.
I20260807 03:13:28.101859 3344 leader_election.cc:302] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bf4008691014405af445b55b83433a8, 9f591de0674940649f693e1779798a8c; no voters:
I20260807 03:13:28.102133 3631 raft_consensus.cc:2781] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:28.102603 3631 raft_consensus.cc:686] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7bf4008691014405af445b55b83433a8, State: Running, Role: LEADER
I20260807 03:13:28.103084 3631 consensus_queue.cc:237] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf4008691014405af445b55b83433a8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41613 } } peers { permanent_uuid: "e7c840b99e254d648945bddf74c6cfc8" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38163 } } peers { permanent_uuid: "9f591de0674940649f693e1779798a8c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45089 } }
I20260807 03:13:28.107765 3121 catalog_manager.cc:5495] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bf4008691014405af445b55b83433a8 (127.17.31.2). New cstate: current_term: 1 leader_uuid: "7bf4008691014405af445b55b83433a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf4008691014405af445b55b83433a8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7c840b99e254d648945bddf74c6cfc8" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f591de0674940649f693e1779798a8c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45089 } health_report { overall_health: UNKNOWN } } }
I20260807 03:13:28.135885 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:28.130219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39504 (local address 127.17.31.2:41613)
0807 03:13:28.130430 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:28.130434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:28.130454 (+ 20us) server_negotiation.cc:407] Connection header received
0807 03:13:28.130663 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:28.130667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:28.130745 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:28.130867 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:13:28.131228 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:28.131984 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:28.133626 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:28.134039 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:28.134141 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:28.134734 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:13:28.135252 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:13:28.135374 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:28.135458 (+ 84us) server_negotiation.cc:299] Negotiation successful
0807 03:13:28.135522 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:13:28.143779 3556 raft_consensus.cc:1261] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c [term 1 FOLLOWER]: Refusing update from remote peer 7bf4008691014405af445b55b83433a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:13:28.143991 3269 raft_consensus.cc:1261] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8 [term 1 FOLLOWER]: Refusing update from remote peer 7bf4008691014405af445b55b83433a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:13:28.144819 3636 consensus_queue.cc:1035] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f591de0674940649f693e1779798a8c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:13:28.145175 3631 consensus_queue.cc:1035] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7c840b99e254d648945bddf74c6cfc8" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:13:28.171511 3121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:13:28.171797 3121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:13:28.175397 3121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56bb9be6758148d4a0fdb6cff6f0d0e0: Sending DeleteTablet for 3 replicas of tablet 44967480e6804834b261561f8b68fbb6
I20260807 03:13:28.176445 3393 tablet_service.cc:1514] Processing DeleteTablet for tablet 44967480e6804834b261561f8b68fbb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:28 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:39488
I20260807 03:13:28.176549 3536 tablet_service.cc:1514] Processing DeleteTablet for tablet 44967480e6804834b261561f8b68fbb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:28 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:33610
I20260807 03:13:28.176607 3248 tablet_service.cc:1514] Processing DeleteTablet for tablet 44967480e6804834b261561f8b68fbb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:28 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:60938
I20260807 03:13:28.177742 3649 tablet_replica.cc:330] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8: stopping tablet replica
I20260807 03:13:28.178190 3649 raft_consensus.cc:2229] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:13:28.178591 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 3184
I20260807 03:13:28.179055 3649 raft_consensus.cc:2258] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:28.181336 3649 ts_tablet_manager.cc:1905] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:28.188903 3649 ts_tablet_manager.cc:1918] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:28.189124 3649 log.cc:1196] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/44967480e6804834b261561f8b68fbb6
I20260807 03:13:28.189823 3649 ts_tablet_manager.cc:1939] T 44967480e6804834b261561f8b68fbb6 P 7bf4008691014405af445b55b83433a8: Deleting consensus metadata
I20260807 03:13:28.191510 3107 catalog_manager.cc:4841] TS 7bf4008691014405af445b55b83433a8 (127.17.31.2:41613): tablet 44967480e6804834b261561f8b68fbb6 (table test-table [id=d2c5c152a1fc4ca4b996b51812f48f42]) successfully deleted
I20260807 03:13:28.192165 3651 tablet_replica.cc:330] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8: stopping tablet replica
I20260807 03:13:28.192392 3650 tablet_replica.cc:330] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c: stopping tablet replica
I20260807 03:13:28.192700 3651 raft_consensus.cc:2229] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:13:28.192785 3650 raft_consensus.cc:2229] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:13:28.193300 3650 raft_consensus.cc:2258] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:28.193285 3651 raft_consensus.cc:2258] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:28.195647 3650 ts_tablet_manager.cc:1905] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:28.195930 3651 ts_tablet_manager.cc:1905] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:28.204586 3651 ts_tablet_manager.cc:1918] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:28.204620 3650 ts_tablet_manager.cc:1918] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:28.204763 3650 log.cc:1196] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/44967480e6804834b261561f8b68fbb6
I20260807 03:13:28.204764 3651 log.cc:1196] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/44967480e6804834b261561f8b68fbb6
I20260807 03:13:28.205331 3650 ts_tablet_manager.cc:1939] T 44967480e6804834b261561f8b68fbb6 P 9f591de0674940649f693e1779798a8c: Deleting consensus metadata
I20260807 03:13:28.205331 3651 ts_tablet_manager.cc:1939] T 44967480e6804834b261561f8b68fbb6 P e7c840b99e254d648945bddf74c6cfc8: Deleting consensus metadata
I20260807 03:13:28.207197 3107 catalog_manager.cc:4841] TS 9f591de0674940649f693e1779798a8c (127.17.31.3:45089): tablet 44967480e6804834b261561f8b68fbb6 (table test-table [id=d2c5c152a1fc4ca4b996b51812f48f42]) successfully deleted
I20260807 03:13:28.207517 3105 catalog_manager.cc:4841] TS e7c840b99e254d648945bddf74c6cfc8 (127.17.31.1:38163): tablet 44967480e6804834b261561f8b68fbb6 (table test-table [id=d2c5c152a1fc4ca4b996b51812f48f42]) successfully deleted
Aug 07 03:13:28 dist-test-slave-tm1g krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:13:28.216563 3620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:28.180939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60950 (local address 127.17.31.1:38163)
0807 03:13:28.181166 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:28.181174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:28.194377 (+ 13203us) server_negotiation.cc:407] Connection header received
0807 03:13:28.194696 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:28.194704 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:28.194802 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:28.194879 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:28.196758 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:28.197951 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:28.199926 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:28.200591 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:28.200699 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:28.210790 (+ 10091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:28.210802 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:28.210815 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:28.210852 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:28.213654 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:28.214403 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:28.214409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:28.214411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:28.214545 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:28.214956 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:28.214960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:28.214962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:28.215302 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:28.215620 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:28.215950 (+ 330us) server_negotiation.cc:299] Negotiation successful
0807 03:13:28.216167 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":68}
I20260807 03:13:28.217183 3311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:28.306398 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 3184
I20260807 03:13:28.332638 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 3325
Aug 07 03:13:28 dist-test-slave-tm1g krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:13:28.350539 3618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:28.333483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39518 (local address 127.17.31.2:41613)
0807 03:13:28.333670 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:28.333674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:28.333772 (+ 98us) server_negotiation.cc:407] Connection header received
0807 03:13:28.333954 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:28.333958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:28.334034 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:28.334114 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:28.335828 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:28.336684 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:28.338616 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:28.339037 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:28.339127 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:28.344318 (+ 5191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:28.344333 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:28.344352 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:28.344391 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:28.347721 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:28.348534 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:28.348541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:28.348543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:28.348724 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:28.349117 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:28.349120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:28.349122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:28.349465 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:28.349717 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:28.350013 (+ 296us) server_negotiation.cc:299] Negotiation successful
0807 03:13:28.350165 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 03:13:28.351382 3457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:28.463228 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 3325
I20260807 03:13:28.486069 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 3470
Aug 07 03:13:28 dist-test-slave-tm1g krb5kdc[3075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:13:28.501084 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:28.486863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.17.31.3:45089)
0807 03:13:28.487059 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:28.487065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:28.487162 (+ 97us) server_negotiation.cc:407] Connection header received
0807 03:13:28.487374 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:28.487379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:28.487467 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:28.487579 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:28.488936 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:28.489687 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:28.491234 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:28.491639 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:28.491771 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:28.496097 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:28.496110 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:28.496127 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:28.496162 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:28.498716 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:28.499383 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:28.499405 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:28.499411 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:28.499581 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:28.499904 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:28.499907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:28.499909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:28.500227 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:28.500450 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:28.500672 (+ 222us) server_negotiation.cc:299] Negotiation successful
0807 03:13:28.500823 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 03:13:28.501735 3597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:28.590545 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 3470
I20260807 03:13:28.609117 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 3089
I20260807 03:13:28.610275 3152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:28.718098 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 3089
2026-08-07T03:13:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4307 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-07T03:13:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:13:28Z Disabled control of system clock
I20260807 03:13:28.798997 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:44143
--webserver_interface=127.17.31.62
--webserver_port=0
--builtin_ntp_servers=127.17.31.20:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:44143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 03:13:29.137513 3660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:29.137777 3660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:29.137856 3660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:29.144662 3660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:13:29.144750 3660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:29.144789 3660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:29.144837 3660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:13:29.144876 3660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:13:29.153420 3660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:44143
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:44143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:29.154603 3660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:29.156308 3660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:29.167937 3665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:29.167831 3668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:29.167989 3666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:29.168869 3660 server_base.cc:1034] running on GCE node
I20260807 03:13:29.169519 3660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:29.170873 3660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:29.172066 3660 hybrid_clock.cc:648] HybridClock initialized: now 1786072409172007 us; error 48 us; skew 500 ppm
I20260807 03:13:29.172413 3660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:13:29.178210 3660 webserver.cc:460] Webserver started at http://127.17.31.62:37949/ using document root <none> and password file <none>
I20260807 03:13:29.178892 3660 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:29.179004 3660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:29.179324 3660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:29.182082 3660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "533a829ea34647bc86b61e64bdb84ad6"
format_stamp: "Formatted at 2026-08-07 03:13:29 on dist-test-slave-tm1g"
server_key: "0c6999560e031f6939c78763c319c3d8"
server_key_iv: "5fcd3c5653afde11d004b1e6b5cf5ac0"
server_key_version: "encryptionkey@0"
I20260807 03:13:29.182926 3660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "533a829ea34647bc86b61e64bdb84ad6"
format_stamp: "Formatted at 2026-08-07 03:13:29 on dist-test-slave-tm1g"
server_key: "0c6999560e031f6939c78763c319c3d8"
server_key_iv: "5fcd3c5653afde11d004b1e6b5cf5ac0"
server_key_version: "encryptionkey@0"
I20260807 03:13:29.187799 3660 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 03:13:29.191098 3674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:29.192382 3660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:13:29.192572 3660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "533a829ea34647bc86b61e64bdb84ad6"
format_stamp: "Formatted at 2026-08-07 03:13:29 on dist-test-slave-tm1g"
server_key: "0c6999560e031f6939c78763c319c3d8"
server_key_iv: "5fcd3c5653afde11d004b1e6b5cf5ac0"
server_key_version: "encryptionkey@0"
I20260807 03:13:29.192790 3660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:29.217111 3660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:29.217854 3660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:29.218137 3660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:29.236217 3660 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:44143
I20260807 03:13:29.236250 3728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:44143 every 8 connection(s)
I20260807 03:13:29.237448 3660 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:13:29.240921 3729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:29.242663 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 3660
I20260807 03:13:29.242844 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:13:29.243288 17532 external_mini_cluster.cc:1428] Setting key 2643b37c2429354313edad49e933e9f2
I20260807 03:13:29.252467 3729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6: Bootstrap starting.
I20260807 03:13:29.255978 3729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:29.256716 3732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:29.245066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44012 (local address 127.17.31.62:44143)
0807 03:13:29.246062 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:29.246072 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:29.246113 (+ 41us) server_negotiation.cc:407] Connection header received
0807 03:13:29.247797 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:29.247817 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:29.248221 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:29.248515 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:29.249121 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:29.250489 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:29.252429 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:29.253106 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:29.253227 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:29.254502 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:29.254510 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:29.254521 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:29.255035 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:29.255120 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:29.255143 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:29.255228 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:29.255466 (+ 238us) server_negotiation.cc:299] Negotiation successful
0807 03:13:29.255761 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":419,"threads_started":1}
I20260807 03:13:29.257082 3729 log.cc:824] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:29.259899 3729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6: No bootstrap required, opened a new log
I20260807 03:13:29.266907 3729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533a829ea34647bc86b61e64bdb84ad6" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44143 } }
I20260807 03:13:29.267208 3729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:29.267277 3729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533a829ea34647bc86b61e64bdb84ad6, State: Initialized, Role: FOLLOWER
I20260807 03:13:29.267760 3729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533a829ea34647bc86b61e64bdb84ad6" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44143 } }
I20260807 03:13:29.267956 3729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:13:29.268054 3729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:13:29.268173 3729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:29.270072 3729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533a829ea34647bc86b61e64bdb84ad6" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44143 } }
I20260807 03:13:29.270700 3729 leader_election.cc:302] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 533a829ea34647bc86b61e64bdb84ad6; no voters:
I20260807 03:13:29.271216 3729 leader_election.cc:288] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:13:29.271364 3735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:29.272068 3735 raft_consensus.cc:686] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [term 1 LEADER]: Becoming Leader. State: Replica: 533a829ea34647bc86b61e64bdb84ad6, State: Running, Role: LEADER
I20260807 03:13:29.272545 3735 consensus_queue.cc:237] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533a829ea34647bc86b61e64bdb84ad6" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44143 } }
I20260807 03:13:29.273154 3729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:13:29.275434 3738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 533a829ea34647bc86b61e64bdb84ad6. Latest consensus state: current_term: 1 leader_uuid: "533a829ea34647bc86b61e64bdb84ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533a829ea34647bc86b61e64bdb84ad6" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44143 } } }
I20260807 03:13:29.275496 3736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "533a829ea34647bc86b61e64bdb84ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533a829ea34647bc86b61e64bdb84ad6" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 44143 } } }
I20260807 03:13:29.275887 3736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:29.275893 3738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6 [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:29.277418 3745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:13:29.282718 3745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:13:29.290674 3745 catalog_manager.cc:1347] Generated new cluster ID: 4e72c73adff24e82a49362c4bfdcecc5
I20260807 03:13:29.290779 3745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:13:29.310086 3745 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:13:29.311311 3745 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:13:29.320437 3745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6: Generated new TSK 0
I20260807 03:13:29.321172 3745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:13:29.328909 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44143
--builtin_ntp_servers=127.17.31.20:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 03:13:29.654230 3755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:29.654532 3755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:29.654700 3755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:29.661569 3755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:29.661651 3755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:29.661837 3755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:13:29.670487 3755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:29.671595 3755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:29.673252 3755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:29.685892 3760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:29.685954 3761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:29.685981 3763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:29.687039 3755 server_base.cc:1034] running on GCE node
I20260807 03:13:29.687583 3755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:29.688450 3755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:29.689697 3755 hybrid_clock.cc:648] HybridClock initialized: now 1786072409689662 us; error 45 us; skew 500 ppm
I20260807 03:13:29.689977 3755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:13:29.692106 3755 webserver.cc:460] Webserver started at http://127.17.31.1:33515/ using document root <none> and password file <none>
I20260807 03:13:29.692747 3755 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:29.692875 3755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:29.693174 3755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:29.695923 3755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "528562e3efcf468391b88d9bc4070e49"
format_stamp: "Formatted at 2026-08-07 03:13:29 on dist-test-slave-tm1g"
server_key: "5c0ac7a444e86f35705053e8239d6062"
server_key_iv: "7b395da12efd8d76c876dca63194eb72"
server_key_version: "encryptionkey@0"
I20260807 03:13:29.696762 3755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "528562e3efcf468391b88d9bc4070e49"
format_stamp: "Formatted at 2026-08-07 03:13:29 on dist-test-slave-tm1g"
server_key: "5c0ac7a444e86f35705053e8239d6062"
server_key_iv: "7b395da12efd8d76c876dca63194eb72"
server_key_version: "encryptionkey@0"
I20260807 03:13:29.701547 3755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260807 03:13:29.704828 3769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:29.706007 3755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 03:13:29.706179 3755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "528562e3efcf468391b88d9bc4070e49"
format_stamp: "Formatted at 2026-08-07 03:13:29 on dist-test-slave-tm1g"
server_key: "5c0ac7a444e86f35705053e8239d6062"
server_key_iv: "7b395da12efd8d76c876dca63194eb72"
server_key_version: "encryptionkey@0"
I20260807 03:13:29.706416 3755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:29.725272 3755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:29.726013 3755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:29.726284 3755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:29.727236 3755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:29.728582 3755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:29.728708 3755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:29.728827 3755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:29.728915 3755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:29.764897 3755 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:46837
I20260807 03:13:29.764979 3884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:46837 every 8 connection(s)
I20260807 03:13:29.766144 3755 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:13:29.767774 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 3755
I20260807 03:13:29.767953 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:13:29.768479 17532 external_mini_cluster.cc:1428] Setting key 7620ed8e6ec2451f5a7a79c209b74a48
I20260807 03:13:29.771744 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44143
--builtin_ntp_servers=127.17.31.20:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 03:13:29.780426 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:29.768890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:59369 (local address 127.17.31.62:44143)
0807 03:13:29.771068 (+ 2178us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:29.771074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:29.771115 (+ 41us) server_negotiation.cc:407] Connection header received
0807 03:13:29.772814 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:29.772823 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:29.772937 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:29.773058 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:29.774250 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:29.775406 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:29.777258 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:29.778031 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:29.778173 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:29.779492 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:29.779501 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:29.779516 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:29.779541 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:29.779591 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:29.779609 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:29.779713 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:29.779904 (+ 191us) server_negotiation.cc:299] Negotiation successful
0807 03:13:29.780008 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2066,"thread_start_us":1964,"threads_started":1}
I20260807 03:13:29.781116 3888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:29.769451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44143 (local address 127.17.31.1:59369)
0807 03:13:29.770824 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:29.770888 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:29.772562 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:29.773294 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:29.773307 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:29.773416 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:29.774056 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:29.774073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:29.775548 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:29.775552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:29.777123 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:29.777136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:29.778229 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:29.778234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:29.778504 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:29.779184 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:13:29.779187 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:29.779222 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:13:29.779295 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:29.779718 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:29.779726 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:29.779732 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 03:13:29.779862 (+ 130us) client_negotiation.cc:240] Negotiation successful
0807 03:13:29.780123 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":390,"threads_started":1}
I20260807 03:13:29.782667 3885 heartbeater.cc:347] Connected to a master server at 127.17.31.62:44143
I20260807 03:13:29.782946 3885 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:29.783660 3885 heartbeater.cc:511] Master 127.17.31.62:44143 requested a full tablet report, sending...
I20260807 03:13:29.785740 3691 ts_manager.cc:194] Registered new tserver with Master: 528562e3efcf468391b88d9bc4070e49 (127.17.31.1:46837)
I20260807 03:13:29.787959 3691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.31.1:59369
W20260807 03:13:30.104955 3889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:30.105198 3889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:30.105280 3889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:30.112391 3889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:30.112478 3889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:30.112582 3889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:13:30.121196 3889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:30.122355 3889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:30.124042 3889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:30.136585 3894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:30.136654 3897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:30.136693 3895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:30.137643 3889 server_base.cc:1034] running on GCE node
I20260807 03:13:30.138254 3889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:30.139151 3889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:30.140388 3889 hybrid_clock.cc:648] HybridClock initialized: now 1786072410140344 us; error 57 us; skew 500 ppm
I20260807 03:13:30.140690 3889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:13:30.142791 3889 webserver.cc:460] Webserver started at http://127.17.31.2:38133/ using document root <none> and password file <none>
I20260807 03:13:30.143458 3889 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:30.143587 3889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:30.143903 3889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:30.146664 3889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "d73c4239011549b88fe4fcc292b55bde"
format_stamp: "Formatted at 2026-08-07 03:13:30 on dist-test-slave-tm1g"
server_key: "c40c58aafdaba3ccddc3b0a1bf63cf64"
server_key_iv: "28ff787b9ec702a29ba7ee8826ad2c02"
server_key_version: "encryptionkey@0"
I20260807 03:13:30.147433 3889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "d73c4239011549b88fe4fcc292b55bde"
format_stamp: "Formatted at 2026-08-07 03:13:30 on dist-test-slave-tm1g"
server_key: "c40c58aafdaba3ccddc3b0a1bf63cf64"
server_key_iv: "28ff787b9ec702a29ba7ee8826ad2c02"
server_key_version: "encryptionkey@0"
I20260807 03:13:30.152415 3889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260807 03:13:30.155622 3903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:30.156746 3889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 03:13:30.156924 3889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "d73c4239011549b88fe4fcc292b55bde"
format_stamp: "Formatted at 2026-08-07 03:13:30 on dist-test-slave-tm1g"
server_key: "c40c58aafdaba3ccddc3b0a1bf63cf64"
server_key_iv: "28ff787b9ec702a29ba7ee8826ad2c02"
server_key_version: "encryptionkey@0"
I20260807 03:13:30.157125 3889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:30.174273 3889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:30.175030 3889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:30.175336 3889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:30.176226 3889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:30.177603 3889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:30.177722 3889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:30.177848 3889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:30.177944 3889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:30.210759 3889 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:46595
I20260807 03:13:30.210839 4020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:46595 every 8 connection(s)
I20260807 03:13:30.212110 3889 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:13:30.221526 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 3889
I20260807 03:13:30.221704 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:13:30.222265 17532 external_mini_cluster.cc:1428] Setting key ee267280d78189e6f7e99a8b9549e54e
I20260807 03:13:30.224226 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:30.214455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:52551 (local address 127.17.31.62:44143)
0807 03:13:30.214632 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:30.214637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:30.216199 (+ 1562us) server_negotiation.cc:407] Connection header received
0807 03:13:30.217934 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:30.217938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:30.218002 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:30.218090 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:30.219191 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.219903 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.221335 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.221953 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.222076 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:30.223439 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:30.223445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:30.223450 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:30.223476 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:30.223506 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:30.223517 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:30.223611 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:30.223829 (+ 218us) server_negotiation.cc:299] Negotiation successful
0807 03:13:30.223923 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:13:30.225006 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44143
--builtin_ntp_servers=127.17.31.20:44881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 03:13:30.225144 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:30.214924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44143 (local address 127.17.31.2:52551)
0807 03:13:30.215976 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:30.216052 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:30.217702 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:30.218278 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:30.218285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:30.218444 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:30.219043 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:30.219060 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.220027 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.220031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:30.221201 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:30.221212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.222176 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.222183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:30.222482 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:30.223131 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:13:30.223135 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:30.223185 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:13:30.223258 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:30.223610 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:30.223622 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:30.223631 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 03:13:30.223796 (+ 165us) client_negotiation.cc:240] Negotiation successful
0807 03:13:30.224146 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":253,"threads_started":1}
I20260807 03:13:30.226549 4021 heartbeater.cc:347] Connected to a master server at 127.17.31.62:44143
I20260807 03:13:30.226845 4021 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:30.227514 4021 heartbeater.cc:511] Master 127.17.31.62:44143 requested a full tablet report, sending...
I20260807 03:13:30.228891 3691 ts_manager.cc:194] Registered new tserver with Master: d73c4239011549b88fe4fcc292b55bde (127.17.31.2:46595)
I20260807 03:13:30.230082 3691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.31.2:52551
W20260807 03:13:30.552559 4024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:30.552800 4024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:30.552949 4024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:30.560002 4024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:30.560112 4024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:30.560283 4024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:13:30.568988 4024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:44881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:44143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:30.570155 4024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:30.571817 4024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:30.584293 4029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:30.584323 4030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:30.584367 4032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:30.585567 4024 server_base.cc:1034] running on GCE node
I20260807 03:13:30.586117 4024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:30.586987 4024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:30.588212 4024 hybrid_clock.cc:648] HybridClock initialized: now 1786072410588164 us; error 58 us; skew 500 ppm
I20260807 03:13:30.588510 4024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:13:30.590595 4024 webserver.cc:460] Webserver started at http://127.17.31.3:34739/ using document root <none> and password file <none>
I20260807 03:13:30.591238 4024 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:30.591377 4024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:30.591670 4024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:30.594384 4024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "ab99f15e9c4a45c9b3540f40832b1405"
format_stamp: "Formatted at 2026-08-07 03:13:30 on dist-test-slave-tm1g"
server_key: "956308e248d1c7429f2044d2cc7ed288"
server_key_iv: "abc1899e4915d85bf77424452214b5ee"
server_key_version: "encryptionkey@0"
I20260807 03:13:30.595165 4024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "ab99f15e9c4a45c9b3540f40832b1405"
format_stamp: "Formatted at 2026-08-07 03:13:30 on dist-test-slave-tm1g"
server_key: "956308e248d1c7429f2044d2cc7ed288"
server_key_iv: "abc1899e4915d85bf77424452214b5ee"
server_key_version: "encryptionkey@0"
I20260807 03:13:30.599993 4024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 03:13:30.603195 4038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:30.604439 4024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 03:13:30.604604 4024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "ab99f15e9c4a45c9b3540f40832b1405"
format_stamp: "Formatted at 2026-08-07 03:13:30 on dist-test-slave-tm1g"
server_key: "956308e248d1c7429f2044d2cc7ed288"
server_key_iv: "abc1899e4915d85bf77424452214b5ee"
server_key_version: "encryptionkey@0"
I20260807 03:13:30.604800 4024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:30.622138 4024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:30.622915 4024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:30.623190 4024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:30.624076 4024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:30.625407 4024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:30.625527 4024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:30.625662 4024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:30.625749 4024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:30.658457 4024 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:45421
I20260807 03:13:30.658540 4154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:45421 every 8 connection(s)
I20260807 03:13:30.659721 4024 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:13:30.664490 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 4024
I20260807 03:13:30.664675 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:13:30.665246 17532 external_mini_cluster.cc:1428] Setting key bf4922c862fbed68b50a6ef8e654f8a2
I20260807 03:13:30.672273 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:30.662316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:35781 (local address 127.17.31.62:44143)
0807 03:13:30.662520 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:30.662524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:30.663788 (+ 1264us) server_negotiation.cc:407] Connection header received
0807 03:13:30.666117 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:30.666125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:30.666217 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:30.666350 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:30.667485 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.668218 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.669708 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.670131 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.670263 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:30.671486 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:30.671491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:30.671494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:30.671511 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:30.671543 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:30.671554 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:30.671647 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:30.671859 (+ 212us) server_negotiation.cc:299] Negotiation successful
0807 03:13:30.671959 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 03:13:30.673233 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:30.662787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:44143 (local address 127.17.31.3:35781)
0807 03:13:30.663638 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:30.663674 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:30.665866 (+ 2192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:30.666587 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:30.666601 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:30.666731 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:30.667303 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:30.667319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.668382 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.668387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:30.669562 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:30.669574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.670395 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.670404 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:30.670606 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:30.671189 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:13:30.671191 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:30.671226 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:13:30.671296 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:30.671645 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:30.671656 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:30.671666 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 03:13:30.671832 (+ 166us) client_negotiation.cc:240] Negotiation successful
0807 03:13:30.672177 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":273,"threads_started":1}
I20260807 03:13:30.674492 4155 heartbeater.cc:347] Connected to a master server at 127.17.31.62:44143
I20260807 03:13:30.674716 4155 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:30.675303 4155 heartbeater.cc:511] Master 127.17.31.62:44143 requested a full tablet report, sending...
I20260807 03:13:30.676551 3691 ts_manager.cc:194] Registered new tserver with Master: ab99f15e9c4a45c9b3540f40832b1405 (127.17.31.3:45421)
I20260807 03:13:30.677367 3691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.31.3:35781
I20260807 03:13:30.680030 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:13:30.689106 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:30.683874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.17.31.62:44143)
0807 03:13:30.684064 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:30.684067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:30.684549 (+ 482us) server_negotiation.cc:407] Connection header received
0807 03:13:30.684665 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:30.684668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:30.684727 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:30.684822 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:30.685246 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.685910 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.687228 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.687609 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.687697 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:30.688596 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:30.688600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:30.688603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:30.688617 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:30.688644 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:30.688654 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:30.688719 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:30.688847 (+ 128us) server_negotiation.cc:299] Negotiation successful
0807 03:13:30.688912 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 03:13:30.692368 3691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:13:30.698951 3691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:13:30.733961 4170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:30.711215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:45421 (local address 127.0.0.1:34822)
0807 03:13:30.723450 (+ 12235us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:30.723473 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:30.723611 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:30.725098 (+ 1487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:30.725105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:30.725136 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:30.725894 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:30.725908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.728469 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.728476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:30.730961 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:30.730972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.732978 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.732984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:30.733220 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:30.733270 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 03:13:30.733399 (+ 129us) client_negotiation.cc:240] Negotiation successful
0807 03:13:30.733593 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12075,"mutex_wait_us":171,"thread_start_us":12474,"threads_started":1}
I20260807 03:13:30.733994 4169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:30.710693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.17.31.3:45421)
0807 03:13:30.724384 (+ 13691us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:30.724400 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:30.724431 (+ 31us) server_negotiation.cc:407] Connection header received
0807 03:13:30.724529 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:30.724539 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:30.724848 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:30.726083 (+ 1235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:30.726397 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.728270 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.731115 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.732442 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.732573 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:30.732854 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:30.733475 (+ 621us) server_negotiation.cc:299] Negotiation successful
0807 03:13:30.733638 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13537,"thread_start_us":13369,"threads_started":1}
I20260807 03:13:30.734150 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:30.710554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40318 (local address 127.17.31.1:46837)
0807 03:13:30.725715 (+ 15161us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:30.725726 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:30.725758 (+ 32us) server_negotiation.cc:407] Connection header received
0807 03:13:30.725854 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:30.725861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:30.726088 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:30.726283 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:30.727130 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.728273 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.731399 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.732715 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.732849 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:30.733072 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:30.733640 (+ 568us) server_negotiation.cc:299] Negotiation successful
0807 03:13:30.733803 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14986,"thread_start_us":14856,"threads_started":1}
I20260807 03:13:30.734741 4167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:30.710525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:46837 (local address 127.0.0.1:40318)
0807 03:13:30.721619 (+ 11094us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:30.721672 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:30.721870 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:30.726326 (+ 4456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:30.726333 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:30.726369 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:30.726850 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:30.726865 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.728423 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.728428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:30.731210 (+ 2782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:30.731225 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.733194 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.733199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:30.733359 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:30.733398 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 03:13:30.733457 (+ 59us) client_negotiation.cc:240] Negotiation successful
0807 03:13:30.734390 (+ 933us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10769,"mutex_wait_us":970,"thread_start_us":11547,"threads_started":1}
I20260807 03:13:30.737394 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:30.724418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:46595 (local address 127.0.0.1:56818)
0807 03:13:30.729028 (+ 4610us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:30.729050 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:30.729193 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:30.729810 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:30.729814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:30.729838 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:30.730457 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:30.730471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.732104 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.732108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:30.734051 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:30.734350 (+ 299us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.736673 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.736681 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:30.736860 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:30.736917 (+ 57us) client_negotiation.cc:769] Sending connection context
0807 03:13:30.736993 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 03:13:30.737111 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4484,"mutex_wait_us":6,"thread_start_us":4312,"threads_started":1}
I20260807 03:13:30.738040 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:30.723396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.17.31.2:46595)
0807 03:13:30.727677 (+ 4281us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:30.727685 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:30.729312 (+ 1627us) server_negotiation.cc:407] Connection header received
0807 03:13:30.729422 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:30.729431 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:30.729621 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:30.729802 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:30.730634 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.731980 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.734574 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.736015 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.736156 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:30.736388 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:30.737645 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0807 03:13:30.737810 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4113,"thread_start_us":4010,"threads_started":1}
I20260807 03:13:30.737834 4088 tablet_service.cc:1467] Processing CreateTablet for tablet e7a75c544365480695928b1ddc439785 (DEFAULT_TABLE table=test-table [id=97765ed7054644668d3b328d44b517e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:30.738454 3819 tablet_service.cc:1467] Processing CreateTablet for tablet e7a75c544365480695928b1ddc439785 (DEFAULT_TABLE table=test-table [id=97765ed7054644668d3b328d44b517e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:30.739459 4088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a75c544365480695928b1ddc439785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:30.739741 3819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a75c544365480695928b1ddc439785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:30.740959 3955 tablet_service.cc:1467] Processing CreateTablet for tablet e7a75c544365480695928b1ddc439785 (DEFAULT_TABLE table=test-table [id=97765ed7054644668d3b328d44b517e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:30.742198 3955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a75c544365480695928b1ddc439785. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:30.752616 4173 tablet_bootstrap.cc:492] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde: Bootstrap starting.
I20260807 03:13:30.756106 4173 tablet_bootstrap.cc:654] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:30.757077 4173 log.cc:824] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:30.759657 4173 tablet_bootstrap.cc:492] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde: No bootstrap required, opened a new log
I20260807 03:13:30.759932 4173 ts_tablet_manager.cc:1397] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260807 03:13:30.759927 4175 tablet_bootstrap.cc:492] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405: Bootstrap starting.
I20260807 03:13:30.760031 4174 tablet_bootstrap.cc:492] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49: Bootstrap starting.
I20260807 03:13:30.763358 4175 tablet_bootstrap.cc:654] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:30.763963 4174 tablet_bootstrap.cc:654] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:30.764667 4175 log.cc:824] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:30.765272 4174 log.cc:824] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:30.772406 4173 raft_consensus.cc:348] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab99f15e9c4a45c9b3540f40832b1405" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45421 } } peers { permanent_uuid: "528562e3efcf468391b88d9bc4070e49" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46837 } } peers { permanent_uuid: "d73c4239011549b88fe4fcc292b55bde" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 46595 } }
I20260807 03:13:30.773068 4173 raft_consensus.cc:374] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:30.773329 4173 raft_consensus.cc:729] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73c4239011549b88fe4fcc292b55bde, State: Initialized, Role: FOLLOWER
I20260807 03:13:30.774271 4173 consensus_queue.cc:260] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab99f15e9c4a45c9b3540f40832b1405" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45421 } } peers { permanent_uuid: "528562e3efcf468391b88d9bc4070e49" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46837 } } peers { permanent_uuid: "d73c4239011549b88fe4fcc292b55bde" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 46595 } }
I20260807 03:13:30.777702 4175 tablet_bootstrap.cc:492] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405: No bootstrap required, opened a new log
I20260807 03:13:30.778007 4175 ts_tablet_manager.cc:1397] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.010s
I20260807 03:13:30.779515 4174 tablet_bootstrap.cc:492] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49: No bootstrap required, opened a new log
I20260807 03:13:30.779937 4174 ts_tablet_manager.cc:1397] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.012s
I20260807 03:13:30.781769 4021 heartbeater.cc:503] Master 127.17.31.62:44143 was elected leader, sending a full tablet report...
I20260807 03:13:30.782574 4173 ts_tablet_manager.cc:1428] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde: Time spent starting tablet: real 0.023s user 0.013s sys 0.008s
I20260807 03:13:30.789970 4175 raft_consensus.cc:348] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab99f15e9c4a45c9b3540f40832b1405" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45421 } } peers { permanent_uuid: "528562e3efcf468391b88d9bc4070e49" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46837 } } peers { permanent_uuid: "d73c4239011549b88fe4fcc292b55bde" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 46595 } }
I20260807 03:13:30.790405 4175 raft_consensus.cc:374] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:30.790504 4175 raft_consensus.cc:729] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab99f15e9c4a45c9b3540f40832b1405, State: Initialized, Role: FOLLOWER
I20260807 03:13:30.790758 3885 heartbeater.cc:503] Master 127.17.31.62:44143 was elected leader, sending a full tablet report...
I20260807 03:13:30.791074 4175 consensus_queue.cc:260] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab99f15e9c4a45c9b3540f40832b1405" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45421 } } peers { permanent_uuid: "528562e3efcf468391b88d9bc4070e49" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46837 } } peers { permanent_uuid: "d73c4239011549b88fe4fcc292b55bde" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 46595 } }
I20260807 03:13:30.791446 4174 raft_consensus.cc:348] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab99f15e9c4a45c9b3540f40832b1405" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45421 } } peers { permanent_uuid: "528562e3efcf468391b88d9bc4070e49" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46837 } } peers { permanent_uuid: "d73c4239011549b88fe4fcc292b55bde" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 46595 } }
I20260807 03:13:30.791955 4174 raft_consensus.cc:374] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:30.792097 4174 raft_consensus.cc:729] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528562e3efcf468391b88d9bc4070e49, State: Initialized, Role: FOLLOWER
I20260807 03:13:30.792156 4155 heartbeater.cc:503] Master 127.17.31.62:44143 was elected leader, sending a full tablet report...
I20260807 03:13:30.792814 4175 ts_tablet_manager.cc:1428] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405: Time spent starting tablet: real 0.015s user 0.004s sys 0.008s
I20260807 03:13:30.792680 4174 consensus_queue.cc:260] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab99f15e9c4a45c9b3540f40832b1405" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45421 } } peers { permanent_uuid: "528562e3efcf468391b88d9bc4070e49" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46837 } } peers { permanent_uuid: "d73c4239011549b88fe4fcc292b55bde" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 46595 } }
I20260807 03:13:30.794512 4174 ts_tablet_manager.cc:1428] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
W20260807 03:13:30.911602 4156 tablet.cc:2369] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:13:30.936250 4180 raft_consensus.cc:482] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:13:30.936558 4180 raft_consensus.cc:504] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab99f15e9c4a45c9b3540f40832b1405" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45421 } } peers { permanent_uuid: "528562e3efcf468391b88d9bc4070e49" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46837 } } peers { permanent_uuid: "d73c4239011549b88fe4fcc292b55bde" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 46595 } }
I20260807 03:13:30.938238 4180 leader_election.cc:288] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 528562e3efcf468391b88d9bc4070e49 (127.17.31.1:46837), d73c4239011549b88fe4fcc292b55bde (127.17.31.2:46595)
I20260807 03:13:30.944850 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:30.938564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:46837 (local address 127.17.31.3:51015)
0807 03:13:30.938764 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:30.938786 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:30.938930 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:30.939521 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:30.939525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:30.939563 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:30.939809 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:30.939820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.941051 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.941056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:30.942674 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:30.942686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.944011 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.944018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:30.944219 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:30.944277 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 03:13:30.944368 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 03:13:30.944510 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 03:13:30.945240 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:30.938877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:51015 (local address 127.17.31.1:46837)
0807 03:13:30.939117 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:30.939125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:30.939149 (+ 24us) server_negotiation.cc:407] Connection header received
0807 03:13:30.939236 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:30.939244 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:30.939356 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:30.939463 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:30.939985 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.940919 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.942834 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.943859 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.943953 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:30.944048 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:30.944467 (+ 419us) server_negotiation.cc:299] Negotiation successful
0807 03:13:30.944686 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 03:13:30.945729 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:30.939202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:51891 (local address 127.17.31.2:46595)
0807 03:13:30.939409 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:30.939414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:30.939651 (+ 237us) server_negotiation.cc:407] Connection header received
0807 03:13:30.939809 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:30.939815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:30.939921 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:30.940006 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:30.940582 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.941530 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.943501 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.944546 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.944661 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:30.944759 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:30.945271 (+ 512us) server_negotiation.cc:299] Negotiation successful
0807 03:13:30.945454 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 03:13:30.946484 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:30.938900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:46595 (local address 127.17.31.3:51891)
0807 03:13:30.939546 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:30.939564 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:30.939687 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:30.940039 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:30.940045 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:30.940073 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:30.940426 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:30.940439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.941677 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.941680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:30.943318 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:30.943332 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:30.944878 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:30.944885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:30.945081 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:30.945119 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 03:13:30.945969 (+ 850us) client_negotiation.cc:240] Negotiation successful
0807 03:13:30.946126 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":3,"thread_start_us":315,"threads_started":1}
I20260807 03:13:30.946247 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a75c544365480695928b1ddc439785" candidate_uuid: "ab99f15e9c4a45c9b3540f40832b1405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528562e3efcf468391b88d9bc4070e49" is_pre_election: true
I20260807 03:13:30.946964 3840 raft_consensus.cc:2454] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab99f15e9c4a45c9b3540f40832b1405 in term 0.
I20260807 03:13:30.947672 3975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a75c544365480695928b1ddc439785" candidate_uuid: "ab99f15e9c4a45c9b3540f40832b1405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73c4239011549b88fe4fcc292b55bde" is_pre_election: true
I20260807 03:13:30.947862 4042 leader_election.cc:302] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528562e3efcf468391b88d9bc4070e49, ab99f15e9c4a45c9b3540f40832b1405; no voters:
I20260807 03:13:30.948222 3975 raft_consensus.cc:2454] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab99f15e9c4a45c9b3540f40832b1405 in term 0.
I20260807 03:13:30.948266 4180 raft_consensus.cc:2781] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:13:30.948415 4180 raft_consensus.cc:482] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:13:30.948529 4180 raft_consensus.cc:3037] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:30.950779 4180 raft_consensus.cc:504] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab99f15e9c4a45c9b3540f40832b1405" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45421 } } peers { permanent_uuid: "528562e3efcf468391b88d9bc4070e49" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46837 } } peers { permanent_uuid: "d73c4239011549b88fe4fcc292b55bde" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 46595 } }
I20260807 03:13:30.951313 4180 leader_election.cc:288] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [CANDIDATE]: Term 1 election: Requested vote from peers 528562e3efcf468391b88d9bc4070e49 (127.17.31.1:46837), d73c4239011549b88fe4fcc292b55bde (127.17.31.2:46595)
I20260807 03:13:30.951761 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a75c544365480695928b1ddc439785" candidate_uuid: "ab99f15e9c4a45c9b3540f40832b1405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528562e3efcf468391b88d9bc4070e49"
I20260807 03:13:30.951830 3975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a75c544365480695928b1ddc439785" candidate_uuid: "ab99f15e9c4a45c9b3540f40832b1405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73c4239011549b88fe4fcc292b55bde"
I20260807 03:13:30.951937 3840 raft_consensus.cc:3037] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:30.952034 3975 raft_consensus.cc:3037] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:30.954139 3840 raft_consensus.cc:2454] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab99f15e9c4a45c9b3540f40832b1405 in term 1.
I20260807 03:13:30.954241 3975 raft_consensus.cc:2454] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab99f15e9c4a45c9b3540f40832b1405 in term 1.
I20260807 03:13:30.954576 4042 leader_election.cc:302] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528562e3efcf468391b88d9bc4070e49, ab99f15e9c4a45c9b3540f40832b1405; no voters:
I20260807 03:13:30.954833 4180 raft_consensus.cc:2781] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:30.955235 4180 raft_consensus.cc:686] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [term 1 LEADER]: Becoming Leader. State: Replica: ab99f15e9c4a45c9b3540f40832b1405, State: Running, Role: LEADER
I20260807 03:13:30.955672 4180 consensus_queue.cc:237] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab99f15e9c4a45c9b3540f40832b1405" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45421 } } peers { permanent_uuid: "528562e3efcf468391b88d9bc4070e49" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46837 } } peers { permanent_uuid: "d73c4239011549b88fe4fcc292b55bde" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 46595 } }
I20260807 03:13:30.960126 3690 catalog_manager.cc:5495] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab99f15e9c4a45c9b3540f40832b1405 (127.17.31.3). New cstate: current_term: 1 leader_uuid: "ab99f15e9c4a45c9b3540f40832b1405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab99f15e9c4a45c9b3540f40832b1405" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 45421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "528562e3efcf468391b88d9bc4070e49" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d73c4239011549b88fe4fcc292b55bde" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 46595 } health_report { overall_health: UNKNOWN } } }
W20260807 03:13:30.964047 4022 tablet.cc:2369] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:13:31.018549 3886 tablet.cc:2369] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:13:31.027153 4169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:31.022061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34838 (local address 127.17.31.3:45421)
0807 03:13:31.022238 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:31.022243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:31.022260 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:13:31.022439 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:31.022443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:31.022528 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:31.022646 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:13:31.022974 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:31.023721 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:31.025168 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:31.025575 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:31.025662 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:31.026092 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:13:31.026625 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:13:31.026729 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:31.026826 (+ 97us) server_negotiation.cc:299] Negotiation successful
0807 03:13:31.026883 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:13:31.035224 3975 raft_consensus.cc:1261] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde [term 1 FOLLOWER]: Refusing update from remote peer ab99f15e9c4a45c9b3540f40832b1405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:13:31.035224 3840 raft_consensus.cc:1261] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49 [term 1 FOLLOWER]: Refusing update from remote peer ab99f15e9c4a45c9b3540f40832b1405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:13:31.036170 4184 consensus_queue.cc:1035] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73c4239011549b88fe4fcc292b55bde" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 46595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:13:31.036562 4180 consensus_queue.cc:1035] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "528562e3efcf468391b88d9bc4070e49" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:13:31.053318 4189 mvcc.cc:205] Tried to move back new op lower bound from 7315752595589160960 to 7315752595277905920. Current Snapshot: MvccSnapshot[applied={T|T < 7315752595589160960}]
I20260807 03:13:31.058773 3691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:13:31.059053 3691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:13:31.062700 3691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 533a829ea34647bc86b61e64bdb84ad6: Sending DeleteTablet for 3 replicas of tablet e7a75c544365480695928b1ddc439785
I20260807 03:13:31.063755 4088 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a75c544365480695928b1ddc439785 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:31 UTC) from {username='slave'} at 127.0.0.1:34822
I20260807 03:13:31.063848 3955 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a75c544365480695928b1ddc439785 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:31 UTC) from {username='slave'} at 127.0.0.1:56818
I20260807 03:13:31.064411 3819 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a75c544365480695928b1ddc439785 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:31 UTC) from {username='slave'} at 127.0.0.1:40318
I20260807 03:13:31.064786 4196 tablet_replica.cc:330] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405: stopping tablet replica
I20260807 03:13:31.065330 4196 raft_consensus.cc:2229] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:13:31.066156 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 3755
I20260807 03:13:31.066984 4196 raft_consensus.cc:2258] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:31.069531 4196 ts_tablet_manager.cc:1905] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:31.073649 4198 tablet_replica.cc:330] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49: stopping tablet replica
I20260807 03:13:31.074239 4198 raft_consensus.cc:2229] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:13:31.074908 4198 raft_consensus.cc:2258] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:31.080132 4197 tablet_replica.cc:330] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde: stopping tablet replica
I20260807 03:13:31.080528 4197 raft_consensus.cc:2229] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:13:31.081004 4197 raft_consensus.cc:2258] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:31.081890 4198 ts_tablet_manager.cc:1905] T e7a75c544365480695928b1ddc439785 P 528562e3efcf468391b88d9bc4070e49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:31.083140 4197 ts_tablet_manager.cc:1905] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:31.084668 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:31.068098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.17.31.1:46837)
0807 03:13:31.068288 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:31.068297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:31.076515 (+ 8218us) server_negotiation.cc:407] Connection header received
0807 03:13:31.076705 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:31.076711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:31.076807 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:31.076920 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:31.077667 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:31.078856 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:31.080758 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:31.081432 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:31.081528 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:31.082964 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:31.082974 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:31.082989 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:31.083732 (+ 743us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:31.083787 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:31.083815 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:31.083903 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:31.084161 (+ 258us) server_negotiation.cc:299] Negotiation successful
0807 03:13:31.084277 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 03:13:31.085780 3880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:31.088434 4196 ts_tablet_manager.cc:1918] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:31.088591 4196 log.cc:1196] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/e7a75c544365480695928b1ddc439785
I20260807 03:13:31.089102 4196 ts_tablet_manager.cc:1939] T e7a75c544365480695928b1ddc439785 P ab99f15e9c4a45c9b3540f40832b1405: Deleting consensus metadata
I20260807 03:13:31.090898 3675 catalog_manager.cc:4841] TS ab99f15e9c4a45c9b3540f40832b1405 (127.17.31.3:45421): tablet e7a75c544365480695928b1ddc439785 (table test-table [id=97765ed7054644668d3b328d44b517e5]) successfully deleted
I20260807 03:13:31.091053 4197 ts_tablet_manager.cc:1918] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:31.091208 4197 log.cc:1196] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/e7a75c544365480695928b1ddc439785
I20260807 03:13:31.091640 4197 ts_tablet_manager.cc:1939] T e7a75c544365480695928b1ddc439785 P d73c4239011549b88fe4fcc292b55bde: Deleting consensus metadata
I20260807 03:13:31.093044 3678 catalog_manager.cc:4841] TS d73c4239011549b88fe4fcc292b55bde (127.17.31.2:46595): tablet e7a75c544365480695928b1ddc439785 (table test-table [id=97765ed7054644668d3b328d44b517e5]) successfully deleted
I20260807 03:13:31.191998 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 3755
W20260807 03:13:31.211108 3678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.31.1:46837 (error 108)
I20260807 03:13:31.211385 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 3889
I20260807 03:13:31.213640 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:31.213212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:46837 (local address 127.0.0.1:40326)
0807 03:13:31.213392 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:31.213510 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:46837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 03:13:31.213918 3678 catalog_manager.cc:4584] TS 528562e3efcf468391b88d9bc4070e49 (127.17.31.1:46837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7a75c544365480695928b1ddc439785: Network error: Client connection negotiation failed: client connection to 127.17.31.1:46837: connect: Connection refused (error 111)
I20260807 03:13:31.221004 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:31.212717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.17.31.2:46595)
0807 03:13:31.212918 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:31.212924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:31.212945 (+ 21us) server_negotiation.cc:407] Connection header received
0807 03:13:31.213006 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:31.213011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:31.213119 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:31.213224 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:31.213895 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:31.215302 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:31.217181 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:31.217871 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:31.218003 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:31.219236 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:31.219247 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:31.219263 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:31.220033 (+ 770us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:31.220091 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:31.220115 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:31.220233 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:31.220388 (+ 155us) server_negotiation.cc:299] Negotiation successful
0807 03:13:31.220497 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 03:13:31.221751 4016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:31.267395 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:31.266971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:46837 (local address 127.0.0.1:40336)
0807 03:13:31.267180 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:31.267264 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:46837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260807 03:13:31.268436 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:31.268104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:46837 (local address 127.0.0.1:40350)
0807 03:13:31.268242 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:31.268317 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:46837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 03:13:31.329285 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:31.328695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:46837 (local address 127.0.0.1:40364)
0807 03:13:31.328964 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:31.329076 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:46837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260807 03:13:31.330547 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:31.330107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:46837 (local address 127.0.0.1:40374)
0807 03:13:31.330280 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:31.330403 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:46837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 03:13:31.339367 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 3889
I20260807 03:13:31.360001 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 4024
I20260807 03:13:31.368116 4169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:31.360939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.17.31.3:45421)
0807 03:13:31.361104 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:31.361108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:31.361126 (+ 18us) server_negotiation.cc:407] Connection header received
0807 03:13:31.361355 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:31.361360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:31.361434 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:31.361517 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:31.362061 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:31.362909 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:31.364863 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:31.365307 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:31.365396 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:31.366711 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:31.366719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:31.366733 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:31.367335 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:31.367378 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:31.367397 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:31.367482 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:31.367658 (+ 176us) server_negotiation.cc:299] Negotiation successful
0807 03:13:31.367733 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 03:13:31.368875 4150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:31.411238 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:31.410778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:46837 (local address 127.0.0.1:40382)
0807 03:13:31.410985 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:31.411070 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:46837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 03:13:31.412256 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:31.411878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:46837 (local address 127.0.0.1:40396)
0807 03:13:31.412087 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:31.412153 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.1:46837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260807 03:13:31.487913 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 4024
I20260807 03:13:31.509100 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 3660
I20260807 03:13:31.510263 3724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:31.618083 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 3660
2026-08-07T03:13:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2897 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:13:31 dist-test-slave-tm1g krb5kdc[4208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:13:31 dist-test-slave-tm1g krb5kdc[4208](info): set up 2 sockets
Aug 07 03:13:31 dist-test-slave-tm1g krb5kdc[4208](info): commencing operation
krb5kdc: starting...
W20260807 03:13:32.744341 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:13:32 dist-test-slave-tm1g krb5kdc[4208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:13:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:13:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:13:32.997247 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:46649
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:46649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:33.318485 4223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:33.318811 4223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:33.318969 4223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:33.325932 4223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:13:33.326027 4223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:33.326118 4223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:33.326205 4223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:13:33.326293 4223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:13:33.335762 4223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:46649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:46649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:33.337052 4223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:33.338366 4223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:33.350026 4228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:33.350147 4231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:33.350167 4229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:33.351089 4223 server_base.cc:1034] running on GCE node
I20260807 03:13:33.351728 4223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:33.352981 4223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:33.354182 4223 hybrid_clock.cc:648] HybridClock initialized: now 1786072413354129 us; error 49 us; skew 500 ppm
I20260807 03:13:33.354537 4223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:33 dist-test-slave-tm1g krb5kdc[4208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072413, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:33.357473 4223 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:13:33.358960 4223 webserver.cc:460] Webserver started at http://127.17.31.62:41889/ using document root <none> and password file <none>
I20260807 03:13:33.359614 4223 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:33.359756 4223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:33.360118 4223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:33.363088 4223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "3714b30059c446df871b161e6c07e1de"
format_stamp: "Formatted at 2026-08-07 03:13:33 on dist-test-slave-tm1g"
server_key: "6ac6482300653c8cea9ac25cffe825f8"
server_key_iv: "273a1db2639b635142cd43ebe6afb2b4"
server_key_version: "encryptionkey@0"
I20260807 03:13:33.363816 4223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "3714b30059c446df871b161e6c07e1de"
format_stamp: "Formatted at 2026-08-07 03:13:33 on dist-test-slave-tm1g"
server_key: "6ac6482300653c8cea9ac25cffe825f8"
server_key_iv: "273a1db2639b635142cd43ebe6afb2b4"
server_key_version: "encryptionkey@0"
I20260807 03:13:33.368659 4223 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 03:13:33.371886 4238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:33.373078 4223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 03:13:33.373248 4223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "3714b30059c446df871b161e6c07e1de"
format_stamp: "Formatted at 2026-08-07 03:13:33 on dist-test-slave-tm1g"
server_key: "6ac6482300653c8cea9ac25cffe825f8"
server_key_iv: "273a1db2639b635142cd43ebe6afb2b4"
server_key_version: "encryptionkey@0"
I20260807 03:13:33.373430 4223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:33.389791 4223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:33.393471 4223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:33.393751 4223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:33.412190 4223 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:46649
I20260807 03:13:33.412237 4290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:46649 every 8 connection(s)
I20260807 03:13:33.413479 4223 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:13:33.417119 4291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:33.418375 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 4223
I20260807 03:13:33.418568 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:13:33.419138 17532 external_mini_cluster.cc:1428] Setting key 40ec62092a4f16a6c0b0e876d5c20fd2
I20260807 03:13:33.430847 4291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de: Bootstrap starting.
Aug 07 03:13:33 dist-test-slave-tm1g krb5kdc[4208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:13:33.434530 4291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:33.435856 4291 log.cc:824] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:33.439046 4291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de: No bootstrap required, opened a new log
I20260807 03:13:33.443259 4294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:33.420879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.17.31.62:46649)
0807 03:13:33.421662 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:33.421673 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:33.421716 (+ 43us) server_negotiation.cc:407] Connection header received
0807 03:13:33.423327 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:33.423349 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:33.423733 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:33.424043 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:33.425806 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:33.427137 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:33.429130 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:33.429767 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:33.429889 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:33.434736 (+ 4847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:33.434747 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:33.434763 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:33.434813 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:33.438798 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:33.439536 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:33.439544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:33.439552 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:33.439755 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:33.440118 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:33.440123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:33.440126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:33.440802 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:33.441089 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:33.441520 (+ 431us) server_negotiation.cc:299] Negotiation successful
0807 03:13:33.441943 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":379,"threads_started":1}
I20260807 03:13:33.448556 4291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3714b30059c446df871b161e6c07e1de" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46649 } }
I20260807 03:13:33.448861 4291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:33.448939 4291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3714b30059c446df871b161e6c07e1de, State: Initialized, Role: FOLLOWER
I20260807 03:13:33.449465 4291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3714b30059c446df871b161e6c07e1de" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46649 } }
I20260807 03:13:33.449673 4291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:13:33.449769 4291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:13:33.449895 4291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:33.451939 4291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3714b30059c446df871b161e6c07e1de" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46649 } }
I20260807 03:13:33.452338 4291 leader_election.cc:302] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3714b30059c446df871b161e6c07e1de; no voters:
I20260807 03:13:33.453027 4291 leader_election.cc:288] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:13:33.453152 4296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:33.453848 4296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [term 1 LEADER]: Becoming Leader. State: Replica: 3714b30059c446df871b161e6c07e1de, State: Running, Role: LEADER
I20260807 03:13:33.454396 4296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3714b30059c446df871b161e6c07e1de" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46649 } }
I20260807 03:13:33.455009 4291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:13:33.457026 4298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3714b30059c446df871b161e6c07e1de. Latest consensus state: current_term: 1 leader_uuid: "3714b30059c446df871b161e6c07e1de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3714b30059c446df871b161e6c07e1de" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46649 } } }
I20260807 03:13:33.457499 4298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:33.457055 4297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3714b30059c446df871b161e6c07e1de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3714b30059c446df871b161e6c07e1de" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46649 } } }
I20260807 03:13:33.458254 4297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:33.459257 4304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:13:33.463284 4304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:13:33.471045 4304 catalog_manager.cc:1347] Generated new cluster ID: cea9318aed6446fab935341674639bf2
I20260807 03:13:33.471144 4304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:13:33.481914 4304 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:13:33.483179 4304 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:13:33.489383 4304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de: Generated new TSK 0
I20260807 03:13:33.490077 4304 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:13:33.553503 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:46649
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:33.885663 4318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:33.885912 4318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:33.886006 4318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:33.893014 4318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:33.893112 4318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:33.893222 4318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:13:33.902693 4318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:46649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:33.903790 4318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:33.905015 4318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:33.917924 4323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:33.917897 4324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:33.917984 4326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:33.918839 4318 server_base.cc:1034] running on GCE node
I20260807 03:13:33.919394 4318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:33.920380 4318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:33.921605 4318 hybrid_clock.cc:648] HybridClock initialized: now 1786072413921567 us; error 50 us; skew 500 ppm
I20260807 03:13:33.921906 4318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:33 dist-test-slave-tm1g krb5kdc[4208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072413, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:33.924881 4318 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:13:33.926241 4318 webserver.cc:460] Webserver started at http://127.17.31.1:39579/ using document root <none> and password file <none>
I20260807 03:13:33.926944 4318 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:33.927043 4318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:33.927294 4318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:33.929962 4318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "738bd97a9f18421d9931de1cae55b403"
format_stamp: "Formatted at 2026-08-07 03:13:33 on dist-test-slave-tm1g"
server_key: "704ee12c8eeb92616c2087ad5e347c8e"
server_key_iv: "3157b241444275ef89a7a80d7fb99a5a"
server_key_version: "encryptionkey@0"
I20260807 03:13:33.930706 4318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "738bd97a9f18421d9931de1cae55b403"
format_stamp: "Formatted at 2026-08-07 03:13:33 on dist-test-slave-tm1g"
server_key: "704ee12c8eeb92616c2087ad5e347c8e"
server_key_iv: "3157b241444275ef89a7a80d7fb99a5a"
server_key_version: "encryptionkey@0"
I20260807 03:13:33.935489 4318 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 03:13:33.938720 4333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:33.939886 4318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:13:33.940061 4318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "738bd97a9f18421d9931de1cae55b403"
format_stamp: "Formatted at 2026-08-07 03:13:33 on dist-test-slave-tm1g"
server_key: "704ee12c8eeb92616c2087ad5e347c8e"
server_key_iv: "3157b241444275ef89a7a80d7fb99a5a"
server_key_version: "encryptionkey@0"
I20260807 03:13:33.940254 4318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:33.954622 4318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:33.960129 4318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:33.960400 4318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:33.961318 4318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:33.962604 4318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:33.962700 4318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:33.962841 4318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:33.962909 4318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:33.996577 4318 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:42027
I20260807 03:13:33.996630 4449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:42027 every 8 connection(s)
I20260807 03:13:33.997843 4318 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:13:34.007992 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 4318
I20260807 03:13:34.008193 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:13:34.008643 17532 external_mini_cluster.cc:1428] Setting key 5a64cb06a4c1b84b460aad87741e56a4
Aug 07 03:13:34 dist-test-slave-tm1g krb5kdc[4208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072413, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:13:34.017874 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:34.000269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:35821 (local address 127.17.31.62:46649)
0807 03:13:34.000707 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:34.000714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:34.001662 (+ 948us) server_negotiation.cc:407] Connection header received
0807 03:13:34.003491 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:34.003496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:34.003570 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:34.003663 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:34.005113 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:34.005858 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:34.007442 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:34.007876 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:34.008044 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:34.012081 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:34.012086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:34.012089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:34.012119 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:34.014877 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:34.015600 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:34.015611 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:34.015613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:34.015728 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:34.016159 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:34.016164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:34.016169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:34.016437 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:34.016623 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:34.017272 (+ 649us) server_negotiation.cc:299] Negotiation successful
0807 03:13:34.017510 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":201,"threads_started":1}
I20260807 03:13:34.018838 4453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:34.000795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:46649 (local address 127.17.31.1:35821)
0807 03:13:34.001503 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:34.001547 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:34.003239 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:34.003861 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:34.003869 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:34.004357 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:34.004960 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:34.004978 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:34.006001 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:34.006005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:34.007271 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:34.007283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:34.008130 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:34.008140 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:34.008441 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:34.009182 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:13:34.009214 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:34.011875 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:34.015033 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:34.015041 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:34.015055 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:34.015472 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:34.015847 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:34.015851 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:34.015855 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:34.016039 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:34.016578 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:34.016603 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:34.016924 (+ 321us) client_negotiation.cc:769] Sending connection context
0807 03:13:34.017169 (+ 245us) client_negotiation.cc:240] Negotiation successful
0807 03:13:34.017662 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":267,"threads_started":1}
I20260807 03:13:34.020321 4450 heartbeater.cc:347] Connected to a master server at 127.17.31.62:46649
I20260807 03:13:34.020623 4450 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:34.021363 4450 heartbeater.cc:511] Master 127.17.31.62:46649 requested a full tablet report, sending...
I20260807 03:13:34.023604 4255 ts_manager.cc:194] Registered new tserver with Master: 738bd97a9f18421d9931de1cae55b403 (127.17.31.1:42027)
I20260807 03:13:34.025866 4255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:35821
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:13:34.067082 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:46649
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:34.395517 4456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:34.395772 4456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:34.395865 4456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:34.402837 4456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:34.402933 4456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:34.403039 4456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:13:34.412348 4456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:46649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:34.413587 4456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:34.414883 4456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:34.429919 4461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:34.429891 4464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:34.429980 4462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:34.430748 4456 server_base.cc:1034] running on GCE node
I20260807 03:13:34.431391 4456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:34.432382 4456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:34.433614 4456 hybrid_clock.cc:648] HybridClock initialized: now 1786072414433577 us; error 60 us; skew 500 ppm
I20260807 03:13:34.433936 4456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:34 dist-test-slave-tm1g krb5kdc[4208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072414, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:34.442104 4456 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:13:34.443682 4456 webserver.cc:460] Webserver started at http://127.17.31.2:35005/ using document root <none> and password file <none>
I20260807 03:13:34.444474 4456 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:34.444617 4456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:34.444952 4456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:34.447852 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "03b6111325534ceaa47a8fac36b6db8e"
format_stamp: "Formatted at 2026-08-07 03:13:34 on dist-test-slave-tm1g"
server_key: "fe89721f268cbd3763eddfccbdb13c8f"
server_key_iv: "f72f132f9aa871faa5e4f64a51385c75"
server_key_version: "encryptionkey@0"
I20260807 03:13:34.448555 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "03b6111325534ceaa47a8fac36b6db8e"
format_stamp: "Formatted at 2026-08-07 03:13:34 on dist-test-slave-tm1g"
server_key: "fe89721f268cbd3763eddfccbdb13c8f"
server_key_iv: "f72f132f9aa871faa5e4f64a51385c75"
server_key_version: "encryptionkey@0"
I20260807 03:13:34.453619 4456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 03:13:34.457020 4471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:34.458392 4456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 03:13:34.458523 4456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "03b6111325534ceaa47a8fac36b6db8e"
format_stamp: "Formatted at 2026-08-07 03:13:34 on dist-test-slave-tm1g"
server_key: "fe89721f268cbd3763eddfccbdb13c8f"
server_key_iv: "f72f132f9aa871faa5e4f64a51385c75"
server_key_version: "encryptionkey@0"
I20260807 03:13:34.458729 4456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:34.476459 4456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:34.480603 4456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:34.480835 4456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:34.481674 4456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:34.482904 4456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:34.482976 4456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:34.483059 4456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:34.483103 4456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:34.516045 4456 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:37885
I20260807 03:13:34.516093 4586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:37885 every 8 connection(s)
I20260807 03:13:34.517251 4456 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:13:34.520546 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 4456
I20260807 03:13:34.520785 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:13:34.521293 17532 external_mini_cluster.cc:1428] Setting key d4a358350ca6971d49c7f5e6979b16a5
Aug 07 03:13:34 dist-test-slave-tm1g krb5kdc[4208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072414, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:13:34.538260 4589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:34.519759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:37237 (local address 127.17.31.62:46649)
0807 03:13:34.520240 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:34.520249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:34.521264 (+ 1015us) server_negotiation.cc:407] Connection header received
0807 03:13:34.523350 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:34.523355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:34.523447 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:34.523557 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:34.524926 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:34.526133 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:34.527705 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:34.528451 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:34.528594 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:34.532258 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:34.532262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:34.532265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:34.532294 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:34.535084 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:34.535747 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:34.535755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:34.535761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:34.535925 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:34.536310 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:34.536318 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:34.536324 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:34.536710 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:34.536859 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:34.537512 (+ 653us) server_negotiation.cc:299] Negotiation successful
0807 03:13:34.537756 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":206,"threads_started":1}
I20260807 03:13:34.538818 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:34.520261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:46649 (local address 127.17.31.2:37237)
0807 03:13:34.521091 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:34.521140 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:34.523009 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:34.523753 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:34.523760 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:34.524219 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:34.524755 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:34.524771 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:34.526281 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:34.526285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:34.527565 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:34.527578 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:34.528644 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:34.528648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:34.528851 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:34.529466 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:13:34.529489 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:34.532047 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:34.535214 (+ 3167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:34.535219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:34.535229 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:34.535618 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:34.536030 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:34.536033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:34.536035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:34.536198 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:34.536832 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:34.536837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:34.537114 (+ 277us) client_negotiation.cc:769] Sending connection context
0807 03:13:34.537367 (+ 253us) client_negotiation.cc:240] Negotiation successful
0807 03:13:34.537714 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":365,"threads_started":1}
I20260807 03:13:34.540205 4587 heartbeater.cc:347] Connected to a master server at 127.17.31.62:46649
I20260807 03:13:34.540493 4587 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:34.541157 4587 heartbeater.cc:511] Master 127.17.31.62:46649 requested a full tablet report, sending...
I20260807 03:13:34.542616 4255 ts_manager.cc:194] Registered new tserver with Master: 03b6111325534ceaa47a8fac36b6db8e (127.17.31.2:37885)
I20260807 03:13:34.543394 4255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:37237
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:13:34.581244 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:46649
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:34.904368 4593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:34.904650 4593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:34.904742 4593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:34.912154 4593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:34.912254 4593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:34.912367 4593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:13:34.921963 4593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:46649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:34.923215 4593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:34.924459 4593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:34.937311 4598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:34.937242 4601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:34.937286 4599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:34.938273 4593 server_base.cc:1034] running on GCE node
I20260807 03:13:34.938887 4593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:34.939708 4593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:34.940927 4593 hybrid_clock.cc:648] HybridClock initialized: now 1786072414940893 us; error 43 us; skew 500 ppm
I20260807 03:13:34.941217 4593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:34 dist-test-slave-tm1g krb5kdc[4208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072414, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:34.944193 4593 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:13:34.945640 4593 webserver.cc:460] Webserver started at http://127.17.31.3:44741/ using document root <none> and password file <none>
I20260807 03:13:34.946278 4593 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:34.946485 4593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:34.946800 4593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:34.949677 4593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "cff35d3b0e7e4d0b9b7ede7badd82a9c"
format_stamp: "Formatted at 2026-08-07 03:13:34 on dist-test-slave-tm1g"
server_key: "b5aa1f68d3682b7f3a9b363159efc993"
server_key_iv: "a361433696ed579f71153d3f5d38d07a"
server_key_version: "encryptionkey@0"
I20260807 03:13:34.950464 4593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "cff35d3b0e7e4d0b9b7ede7badd82a9c"
format_stamp: "Formatted at 2026-08-07 03:13:34 on dist-test-slave-tm1g"
server_key: "b5aa1f68d3682b7f3a9b363159efc993"
server_key_iv: "a361433696ed579f71153d3f5d38d07a"
server_key_version: "encryptionkey@0"
I20260807 03:13:34.955293 4593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 03:13:34.958781 4608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:34.960155 4593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:13:34.960304 4593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "cff35d3b0e7e4d0b9b7ede7badd82a9c"
format_stamp: "Formatted at 2026-08-07 03:13:34 on dist-test-slave-tm1g"
server_key: "b5aa1f68d3682b7f3a9b363159efc993"
server_key_iv: "a361433696ed579f71153d3f5d38d07a"
server_key_version: "encryptionkey@0"
I20260807 03:13:34.960516 4593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:34.980175 4593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:34.984390 4593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:34.984627 4593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:34.985486 4593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:34.986708 4593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:34.986780 4593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:34.986863 4593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:34.986907 4593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:35.017908 4593 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:36843
I20260807 03:13:35.017958 4724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:36843 every 8 connection(s)
I20260807 03:13:35.019196 4593 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:13:35.025031 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 4593
I20260807 03:13:35.025238 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:13:35.025835 17532 external_mini_cluster.cc:1428] Setting key 9f803542f942015510b11c1b73c5e3b9
I20260807 03:13:35.028694 4450 heartbeater.cc:503] Master 127.17.31.62:46649 was elected leader, sending a full tablet report...
Aug 07 03:13:35 dist-test-slave-tm1g krb5kdc[4208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072414, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:13:35.039161 4589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.021631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:44641 (local address 127.17.31.62:46649)
0807 03:13:35.021822 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:35.021827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:35.022979 (+ 1152us) server_negotiation.cc:407] Connection header received
0807 03:13:35.024741 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:35.024746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:35.024818 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:35.024903 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:35.026890 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.028017 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.029806 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.030364 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.030494 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.033686 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:35.033699 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:35.033702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:35.033734 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:35.036140 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:35.036873 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:35.036882 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:35.036886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:35.037034 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:35.037458 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:35.037463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:35.037465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:35.037724 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:35.037908 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:35.038563 (+ 655us) server_negotiation.cc:299] Negotiation successful
0807 03:13:35.038787 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 03:13:35.040055 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.022036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:46649 (local address 127.17.31.3:44641)
0807 03:13:35.022809 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:35.022868 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:35.024463 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:35.025186 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:35.025198 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:35.025924 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:35.026696 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:35.026721 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.028160 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.028164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:35.029667 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:35.029679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.030608 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.030615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:35.030834 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.031430 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:13:35.031455 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:35.033498 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:35.036282 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:35.036290 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:35.036304 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:35.036738 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:35.037144 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:35.037150 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:35.037154 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:35.037329 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:35.037867 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:35.037875 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:35.038188 (+ 313us) client_negotiation.cc:769] Sending connection context
0807 03:13:35.038488 (+ 300us) client_negotiation.cc:240] Negotiation successful
0807 03:13:35.038981 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":281,"threads_started":1}
I20260807 03:13:35.041466 4725 heartbeater.cc:347] Connected to a master server at 127.17.31.62:46649
I20260807 03:13:35.041749 4725 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:35.042488 4725 heartbeater.cc:511] Master 127.17.31.62:46649 requested a full tablet report, sending...
I20260807 03:13:35.043752 4255 ts_manager.cc:194] Registered new tserver with Master: cff35d3b0e7e4d0b9b7ede7badd82a9c (127.17.31.3:36843)
I20260807 03:13:35.044670 4255 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:44641
I20260807 03:13:35.053125 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:13:35.069679 4589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.057212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46324 (local address 127.17.31.62:46649)
0807 03:13:35.057361 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:35.057365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:35.057869 (+ 504us) server_negotiation.cc:407] Connection header received
0807 03:13:35.058005 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:35.058008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:35.058071 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:35.058148 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:35.059576 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.060253 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.061488 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.061939 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.062029 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.064709 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:35.064713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:35.064717 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:35.064745 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:35.067383 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:35.068081 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:35.068085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:35.068087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:35.068204 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:35.068558 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:35.068561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:35.068563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:35.068811 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:35.069009 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:35.069297 (+ 288us) server_negotiation.cc:299] Negotiation successful
0807 03:13:35.069416 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 03:13:35.073272 4255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:13:35.080183 4255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:13:35.116701 4739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.093688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:36843 (local address 127.0.0.1:39984)
0807 03:13:35.108614 (+ 14926us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:35.108630 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:35.108767 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:35.109343 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:35.109348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:35.109371 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:35.109834 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:35.109847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.111856 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.111861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:35.113541 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:35.113556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.115707 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.115713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:35.115900 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.115962 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 03:13:35.116094 (+ 132us) client_negotiation.cc:240] Negotiation successful
0807 03:13:35.116300 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14820,"thread_start_us":14076,"threads_started":1}
I20260807 03:13:35.116842 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.094420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39984 (local address 127.17.31.3:36843)
0807 03:13:35.100325 (+ 5905us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:35.100337 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:35.108749 (+ 8412us) server_negotiation.cc:407] Connection header received
0807 03:13:35.108966 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:35.108973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:35.109174 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:35.109349 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:35.110037 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.111718 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.113690 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.115494 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.115643 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.115884 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:35.116311 (+ 427us) server_negotiation.cc:299] Negotiation successful
0807 03:13:35.116482 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5702,"thread_start_us":5537,"threads_started":1}
I20260807 03:13:35.116866 4737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.091599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:42027 (local address 127.0.0.1:45792)
0807 03:13:35.106069 (+ 14470us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:35.106121 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:35.106355 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:35.108891 (+ 2536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:35.108899 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:35.108926 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:35.109344 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:35.109359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.111490 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.111495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:35.114055 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:35.114069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.116045 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.116054 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:35.116218 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.116270 (+ 52us) client_negotiation.cc:769] Sending connection context
0807 03:13:35.116355 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 03:13:35.116479 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14140,"mutex_wait_us":65,"thread_start_us":14979,"threads_started":1}
I20260807 03:13:35.117463 4738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.091864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.17.31.1:42027)
0807 03:13:35.108232 (+ 16368us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:35.108243 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:35.108270 (+ 27us) server_negotiation.cc:407] Connection header received
0807 03:13:35.108378 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:35.108387 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:35.108656 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:35.108867 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:35.109600 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.111314 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.114243 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.115782 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.115902 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.116133 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:35.116971 (+ 838us) server_negotiation.cc:299] Negotiation successful
0807 03:13:35.117160 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16187,"thread_start_us":16001,"threads_started":1}
I20260807 03:13:35.117764 4741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.091885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:37885 (local address 127.0.0.1:59416)
0807 03:13:35.108090 (+ 16205us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:35.108109 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:35.108891 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:35.109701 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:35.109706 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:35.109729 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:35.110091 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:35.110099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.112008 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.112014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:35.114285 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:35.114342 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.116329 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.116336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:35.116505 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.116536 (+ 31us) client_negotiation.cc:769] Sending connection context
0807 03:13:35.116632 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 03:13:35.116765 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16087,"thread_start_us":16460,"threads_started":1}
I20260807 03:13:35.118026 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.092165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.17.31.2:37885)
0807 03:13:35.107353 (+ 15188us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:35.107361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:35.108203 (+ 842us) server_negotiation.cc:407] Connection header received
0807 03:13:35.109112 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:35.109120 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:35.109337 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:35.109508 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:35.110280 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.111863 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.114511 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.115914 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.116023 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.116186 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:35.117553 (+ 1367us) server_negotiation.cc:299] Negotiation successful
0807 03:13:35.117715 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15014,"thread_start_us":15176,"threads_started":1}
I20260807 03:13:35.120553 4656 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb3438a59244383b1a0c16cb28b4ede (DEFAULT_TABLE table=test-table [id=ba277640d15e4179942aeb454bd46189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:35.121358 4519 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb3438a59244383b1a0c16cb28b4ede (DEFAULT_TABLE table=test-table [id=ba277640d15e4179942aeb454bd46189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:35.121361 4383 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb3438a59244383b1a0c16cb28b4ede (DEFAULT_TABLE table=test-table [id=ba277640d15e4179942aeb454bd46189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:35.121861 4656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb3438a59244383b1a0c16cb28b4ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:35.122668 4519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb3438a59244383b1a0c16cb28b4ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:35.122691 4383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb3438a59244383b1a0c16cb28b4ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:35.133388 4743 tablet_bootstrap.cc:492] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c: Bootstrap starting.
I20260807 03:13:35.136529 4743 tablet_bootstrap.cc:654] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:35.137843 4745 tablet_bootstrap.cc:492] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403: Bootstrap starting.
I20260807 03:13:35.137902 4743 log.cc:824] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:35.139492 4744 tablet_bootstrap.cc:492] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e: Bootstrap starting.
I20260807 03:13:35.141516 4745 tablet_bootstrap.cc:654] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:35.142459 4744 tablet_bootstrap.cc:654] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:35.142745 4745 log.cc:824] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:35.142907 4743 tablet_bootstrap.cc:492] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c: No bootstrap required, opened a new log
I20260807 03:13:35.143159 4743 ts_tablet_manager.cc:1397] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 03:13:35.143687 4744 log.cc:824] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:35.145884 4745 tablet_bootstrap.cc:492] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403: No bootstrap required, opened a new log
I20260807 03:13:35.146188 4745 ts_tablet_manager.cc:1397] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260807 03:13:35.146538 4744 tablet_bootstrap.cc:492] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e: No bootstrap required, opened a new log
I20260807 03:13:35.146786 4744 ts_tablet_manager.cc:1397] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260807 03:13:35.153618 4744 raft_consensus.cc:348] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738bd97a9f18421d9931de1cae55b403" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42027 } } peers { permanent_uuid: "03b6111325534ceaa47a8fac36b6db8e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37885 } } peers { permanent_uuid: "cff35d3b0e7e4d0b9b7ede7badd82a9c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36843 } }
I20260807 03:13:35.153970 4744 raft_consensus.cc:374] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:35.153743 4743 raft_consensus.cc:348] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738bd97a9f18421d9931de1cae55b403" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42027 } } peers { permanent_uuid: "03b6111325534ceaa47a8fac36b6db8e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37885 } } peers { permanent_uuid: "cff35d3b0e7e4d0b9b7ede7badd82a9c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36843 } }
I20260807 03:13:35.154106 4743 raft_consensus.cc:374] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:35.154114 4744 raft_consensus.cc:729] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03b6111325534ceaa47a8fac36b6db8e, State: Initialized, Role: FOLLOWER
I20260807 03:13:35.154207 4743 raft_consensus.cc:729] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff35d3b0e7e4d0b9b7ede7badd82a9c, State: Initialized, Role: FOLLOWER
I20260807 03:13:35.154770 4744 consensus_queue.cc:260] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738bd97a9f18421d9931de1cae55b403" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42027 } } peers { permanent_uuid: "03b6111325534ceaa47a8fac36b6db8e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37885 } } peers { permanent_uuid: "cff35d3b0e7e4d0b9b7ede7badd82a9c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36843 } }
I20260807 03:13:35.154928 4743 consensus_queue.cc:260] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738bd97a9f18421d9931de1cae55b403" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42027 } } peers { permanent_uuid: "03b6111325534ceaa47a8fac36b6db8e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37885 } } peers { permanent_uuid: "cff35d3b0e7e4d0b9b7ede7badd82a9c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36843 } }
I20260807 03:13:35.157550 4587 heartbeater.cc:503] Master 127.17.31.62:46649 was elected leader, sending a full tablet report...
I20260807 03:13:35.157294 4745 raft_consensus.cc:348] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738bd97a9f18421d9931de1cae55b403" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42027 } } peers { permanent_uuid: "03b6111325534ceaa47a8fac36b6db8e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37885 } } peers { permanent_uuid: "cff35d3b0e7e4d0b9b7ede7badd82a9c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36843 } }
I20260807 03:13:35.157768 4745 raft_consensus.cc:374] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:35.157881 4745 raft_consensus.cc:729] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738bd97a9f18421d9931de1cae55b403, State: Initialized, Role: FOLLOWER
I20260807 03:13:35.158209 4744 ts_tablet_manager.cc:1428] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260807 03:13:35.158632 4745 consensus_queue.cc:260] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738bd97a9f18421d9931de1cae55b403" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42027 } } peers { permanent_uuid: "03b6111325534ceaa47a8fac36b6db8e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37885 } } peers { permanent_uuid: "cff35d3b0e7e4d0b9b7ede7badd82a9c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36843 } }
I20260807 03:13:35.159327 4725 heartbeater.cc:503] Master 127.17.31.62:46649 was elected leader, sending a full tablet report...
I20260807 03:13:35.159897 4743 ts_tablet_manager.cc:1428] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20260807 03:13:35.161373 4745 ts_tablet_manager.cc:1428] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20260807 03:13:35.161682 4751 raft_consensus.cc:482] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:13:35.161914 4751 raft_consensus.cc:504] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738bd97a9f18421d9931de1cae55b403" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42027 } } peers { permanent_uuid: "03b6111325534ceaa47a8fac36b6db8e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37885 } } peers { permanent_uuid: "cff35d3b0e7e4d0b9b7ede7badd82a9c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36843 } }
I20260807 03:13:35.165423 4751 leader_election.cc:288] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03b6111325534ceaa47a8fac36b6db8e (127.17.31.2:37885), cff35d3b0e7e4d0b9b7ede7badd82a9c (127.17.31.3:36843)
I20260807 03:13:35.171753 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.164449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:57693 (local address 127.17.31.2:37885)
0807 03:13:35.164672 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:35.164678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:35.165426 (+ 748us) server_negotiation.cc:407] Connection header received
0807 03:13:35.166125 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:35.166132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:35.166217 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:35.166341 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:35.166990 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.167707 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.169727 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.170793 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.170893 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.170958 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:35.171355 (+ 397us) server_negotiation.cc:299] Negotiation successful
0807 03:13:35.171497 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 03:13:35.171845 4752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.164285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:37885 (local address 127.17.31.1:57693)
0807 03:13:35.165275 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:35.165312 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:35.165888 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:35.166397 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:35.166402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:35.166432 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:35.166845 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:35.166855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.167836 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.167843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:35.169530 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:35.169572 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.170998 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.171006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:35.171170 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.171204 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 03:13:35.171301 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 03:13:35.171466 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":654,"threads_started":1}
I20260807 03:13:35.172425 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.165766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:54275 (local address 127.17.31.3:36843)
0807 03:13:35.165947 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:35.165954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:35.166689 (+ 735us) server_negotiation.cc:407] Connection header received
0807 03:13:35.166899 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:35.166904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:35.166969 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:35.167059 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:35.167551 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.168238 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.170430 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.171375 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.171509 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.171606 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:35.172046 (+ 440us) server_negotiation.cc:299] Negotiation successful
0807 03:13:35.172172 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 03:13:35.172468 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.165860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:36843 (local address 127.17.31.1:54275)
0807 03:13:35.166588 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:35.166602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:35.166749 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:35.167072 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:35.167076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:35.167100 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:35.167411 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:35.167419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.168382 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.168387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:35.170275 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:35.170291 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.171620 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.171629 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:35.171798 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.171856 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 03:13:35.172021 (+ 165us) client_negotiation.cc:240] Negotiation successful
0807 03:13:35.172149 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":438,"threads_started":1}
I20260807 03:13:35.172808 4542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb3438a59244383b1a0c16cb28b4ede" candidate_uuid: "738bd97a9f18421d9931de1cae55b403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03b6111325534ceaa47a8fac36b6db8e" is_pre_election: true
I20260807 03:13:35.173367 4542 raft_consensus.cc:2454] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 738bd97a9f18421d9931de1cae55b403 in term 0.
I20260807 03:13:35.173337 4680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb3438a59244383b1a0c16cb28b4ede" candidate_uuid: "738bd97a9f18421d9931de1cae55b403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff35d3b0e7e4d0b9b7ede7badd82a9c" is_pre_election: true
I20260807 03:13:35.173834 4680 raft_consensus.cc:2454] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 738bd97a9f18421d9931de1cae55b403 in term 0.
I20260807 03:13:35.174063 4336 leader_election.cc:302] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03b6111325534ceaa47a8fac36b6db8e, 738bd97a9f18421d9931de1cae55b403; no voters:
I20260807 03:13:35.174455 4751 raft_consensus.cc:2781] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:13:35.174602 4751 raft_consensus.cc:482] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:13:35.174707 4751 raft_consensus.cc:3037] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:35.176833 4751 raft_consensus.cc:504] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738bd97a9f18421d9931de1cae55b403" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42027 } } peers { permanent_uuid: "03b6111325534ceaa47a8fac36b6db8e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37885 } } peers { permanent_uuid: "cff35d3b0e7e4d0b9b7ede7badd82a9c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36843 } }
I20260807 03:13:35.177408 4751 leader_election.cc:288] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [CANDIDATE]: Term 1 election: Requested vote from peers 03b6111325534ceaa47a8fac36b6db8e (127.17.31.2:37885), cff35d3b0e7e4d0b9b7ede7badd82a9c (127.17.31.3:36843)
I20260807 03:13:35.177745 4542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb3438a59244383b1a0c16cb28b4ede" candidate_uuid: "738bd97a9f18421d9931de1cae55b403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03b6111325534ceaa47a8fac36b6db8e"
I20260807 03:13:35.177947 4542 raft_consensus.cc:3037] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:35.177927 4680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb3438a59244383b1a0c16cb28b4ede" candidate_uuid: "738bd97a9f18421d9931de1cae55b403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff35d3b0e7e4d0b9b7ede7badd82a9c"
I20260807 03:13:35.178112 4680 raft_consensus.cc:3037] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:35.181226 4542 raft_consensus.cc:2454] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 738bd97a9f18421d9931de1cae55b403 in term 1.
I20260807 03:13:35.181221 4680 raft_consensus.cc:2454] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 738bd97a9f18421d9931de1cae55b403 in term 1.
I20260807 03:13:35.181627 4335 leader_election.cc:302] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 738bd97a9f18421d9931de1cae55b403, cff35d3b0e7e4d0b9b7ede7badd82a9c; no voters:
I20260807 03:13:35.181973 4751 raft_consensus.cc:2781] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:35.182428 4751 raft_consensus.cc:686] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [term 1 LEADER]: Becoming Leader. State: Replica: 738bd97a9f18421d9931de1cae55b403, State: Running, Role: LEADER
I20260807 03:13:35.182868 4751 consensus_queue.cc:237] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738bd97a9f18421d9931de1cae55b403" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42027 } } peers { permanent_uuid: "03b6111325534ceaa47a8fac36b6db8e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37885 } } peers { permanent_uuid: "cff35d3b0e7e4d0b9b7ede7badd82a9c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36843 } }
I20260807 03:13:35.187147 4255 catalog_manager.cc:5495] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 reported cstate change: term changed from 0 to 1, leader changed from <none> to 738bd97a9f18421d9931de1cae55b403 (127.17.31.1). New cstate: current_term: 1 leader_uuid: "738bd97a9f18421d9931de1cae55b403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738bd97a9f18421d9931de1cae55b403" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 42027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03b6111325534ceaa47a8fac36b6db8e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cff35d3b0e7e4d0b9b7ede7badd82a9c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36843 } health_report { overall_health: UNKNOWN } } }
I20260807 03:13:35.216409 4738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.211202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45800 (local address 127.17.31.1:42027)
0807 03:13:35.211385 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:35.211390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:35.211517 (+ 127us) server_negotiation.cc:407] Connection header received
0807 03:13:35.211678 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:35.211682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:35.211755 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:35.211876 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:13:35.212212 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.212939 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.214277 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.214702 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.214792 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.215441 (+ 649us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:13:35.215909 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:13:35.216034 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:35.216185 (+ 151us) server_negotiation.cc:299] Negotiation successful
0807 03:13:35.216221 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 03:13:35.224280 4680 raft_consensus.cc:1261] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c [term 1 FOLLOWER]: Refusing update from remote peer 738bd97a9f18421d9931de1cae55b403: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:13:35.224305 4542 raft_consensus.cc:1261] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e [term 1 FOLLOWER]: Refusing update from remote peer 738bd97a9f18421d9931de1cae55b403: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:13:35.225191 4751 consensus_queue.cc:1035] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff35d3b0e7e4d0b9b7ede7badd82a9c" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:13:35.225508 4754 consensus_queue.cc:1035] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03b6111325534ceaa47a8fac36b6db8e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 37885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:13:35.236640 4757 mvcc.cc:205] Tried to move back new op lower bound from 7315752612748574720 to 7315752612592398336. Current Snapshot: MvccSnapshot[applied={T|T < 7315752612748574720}]
I20260807 03:13:35.238521 4758 mvcc.cc:205] Tried to move back new op lower bound from 7315752612748574720 to 7315752612592398336. Current Snapshot: MvccSnapshot[applied={T|T < 7315752612748574720}]
I20260807 03:13:35.249686 4255 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:13:35.249979 4255 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:13:35.253616 4255 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3714b30059c446df871b161e6c07e1de: Sending DeleteTablet for 3 replicas of tablet 5cb3438a59244383b1a0c16cb28b4ede
I20260807 03:13:35.254856 4519 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb3438a59244383b1a0c16cb28b4ede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:35 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:59416
I20260807 03:13:35.255079 4383 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb3438a59244383b1a0c16cb28b4ede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:35 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:45792
I20260807 03:13:35.255060 4656 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb3438a59244383b1a0c16cb28b4ede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:35 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:39984
I20260807 03:13:35.259948 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 4318
I20260807 03:13:35.273752 4771 tablet_replica.cc:330] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c: stopping tablet replica
I20260807 03:13:35.274286 4771 raft_consensus.cc:2229] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:13:35.274720 4769 tablet_replica.cc:330] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e: stopping tablet replica
I20260807 03:13:35.274888 4770 tablet_replica.cc:330] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403: stopping tablet replica
I20260807 03:13:35.275064 4771 raft_consensus.cc:2258] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:35.275128 4769 raft_consensus.cc:2229] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:13:35.275241 4770 raft_consensus.cc:2229] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:13:35.275592 4769 raft_consensus.cc:2258] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:35.275983 4770 raft_consensus.cc:2258] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:35.277783 4771 ts_tablet_manager.cc:1905] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:35.277799 4770 ts_tablet_manager.cc:1905] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 03:13:35 dist-test-slave-tm1g krb5kdc[4208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:13:35.284822 4771 ts_tablet_manager.cc:1918] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:35.284875 4769 ts_tablet_manager.cc:1905] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:35.285122 4771 log.cc:1196] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/5cb3438a59244383b1a0c16cb28b4ede
I20260807 03:13:35.285741 4771 ts_tablet_manager.cc:1939] T 5cb3438a59244383b1a0c16cb28b4ede P cff35d3b0e7e4d0b9b7ede7badd82a9c: Deleting consensus metadata
I20260807 03:13:35.285795 4770 ts_tablet_manager.cc:1918] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:35.285951 4770 log.cc:1196] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/5cb3438a59244383b1a0c16cb28b4ede
I20260807 03:13:35.286541 4770 ts_tablet_manager.cc:1939] T 5cb3438a59244383b1a0c16cb28b4ede P 738bd97a9f18421d9931de1cae55b403: Deleting consensus metadata
I20260807 03:13:35.287839 4240 catalog_manager.cc:4841] TS cff35d3b0e7e4d0b9b7ede7badd82a9c (127.17.31.3:36843): tablet 5cb3438a59244383b1a0c16cb28b4ede (table test-table [id=ba277640d15e4179942aeb454bd46189]) successfully deleted
I20260807 03:13:35.288437 4242 catalog_manager.cc:4841] TS 738bd97a9f18421d9931de1cae55b403 (127.17.31.1:42027): tablet 5cb3438a59244383b1a0c16cb28b4ede (table test-table [id=ba277640d15e4179942aeb454bd46189]) successfully deleted
I20260807 03:13:35.292867 4769 ts_tablet_manager.cc:1918] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:35.293033 4769 log.cc:1196] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/5cb3438a59244383b1a0c16cb28b4ede
I20260807 03:13:35.293597 4769 ts_tablet_manager.cc:1939] T 5cb3438a59244383b1a0c16cb28b4ede P 03b6111325534ceaa47a8fac36b6db8e: Deleting consensus metadata
I20260807 03:13:35.295313 4241 catalog_manager.cc:4841] TS 03b6111325534ceaa47a8fac36b6db8e (127.17.31.2:37885): tablet 5cb3438a59244383b1a0c16cb28b4ede (table test-table [id=ba277640d15e4179942aeb454bd46189]) successfully deleted
I20260807 03:13:35.295796 4738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.261283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45814 (local address 127.17.31.1:42027)
0807 03:13:35.261456 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:35.261463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:35.266529 (+ 5066us) server_negotiation.cc:407] Connection header received
0807 03:13:35.267117 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:35.267126 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:35.267228 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:35.267325 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:35.271591 (+ 4266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.273260 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.278114 (+ 4854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.278866 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.278968 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.288195 (+ 9227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:35.288210 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:35.288226 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:35.288264 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:35.292424 (+ 4160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:35.293308 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:35.293319 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:35.293334 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:35.293532 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:35.293995 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:35.294000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:35.294003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:35.294503 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:35.294796 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:35.295134 (+ 338us) server_negotiation.cc:299] Negotiation successful
0807 03:13:35.295395 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 03:13:35.296450 4444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:35.395249 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 4318
I20260807 03:13:35.417157 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 4456
Aug 07 03:13:35 dist-test-slave-tm1g krb5kdc[4208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:13:35.433840 4740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.417957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59424 (local address 127.17.31.2:37885)
0807 03:13:35.418113 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:35.418118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:35.418196 (+ 78us) server_negotiation.cc:407] Connection header received
0807 03:13:35.418373 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:35.418378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:35.418465 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:35.418586 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:35.419843 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.420965 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.422211 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.422899 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.423010 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.428196 (+ 5186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:35.428211 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:35.428226 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:35.428267 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:35.431203 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:35.431998 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:35.432004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:35.432006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:35.432130 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:35.432476 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:35.432479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:35.432481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:35.432819 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:35.433033 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:35.433393 (+ 360us) server_negotiation.cc:299] Negotiation successful
0807 03:13:35.433551 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 03:13:35.434690 4582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:35.523955 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 4456
I20260807 03:13:35.544117 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 4593
Aug 07 03:13:35 dist-test-slave-tm1g krb5kdc[4208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:13:35.559497 4742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:35.544925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.17.31.3:36843)
0807 03:13:35.545110 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:35.545118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:35.545142 (+ 24us) server_negotiation.cc:407] Connection header received
0807 03:13:35.545360 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:35.545372 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:35.545448 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:35.545557 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:35.546990 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.547734 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.549057 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:35.549465 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:35.549563 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:35.553910 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:35.553925 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:35.553943 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:35.553973 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:35.556728 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:35.557408 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:35.557423 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:35.557425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:35.557549 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:35.557912 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:35.557917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:35.557922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:35.558258 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:35.558637 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:35.558951 (+ 314us) server_negotiation.cc:299] Negotiation successful
0807 03:13:35.559177 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 03:13:35.560111 4720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:35.648849 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 4593
I20260807 03:13:35.667896 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 4223
I20260807 03:13:35.669028 4286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:35.783205 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 4223
2026-08-07T03:13:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4165 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-07T03:13:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:13:35Z Disabled control of system clock
I20260807 03:13:35.863878 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:37745
--webserver_interface=127.17.31.62
--webserver_port=0
--builtin_ntp_servers=127.17.31.20:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:37745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 03:13:36.190635 4780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:36.190918 4780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:36.190997 4780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:36.197773 4780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:13:36.197866 4780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:36.197903 4780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:36.197956 4780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:13:36.198000 4780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:13:36.206843 4780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:37745
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:37745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:36.207991 4780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:36.209668 4780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:36.221760 4788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:36.221850 4786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:36.221877 4785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:36.222667 4780 server_base.cc:1034] running on GCE node
I20260807 03:13:36.223405 4780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:36.224746 4780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:36.225975 4780 hybrid_clock.cc:648] HybridClock initialized: now 1786072416225885 us; error 77 us; skew 500 ppm
I20260807 03:13:36.226389 4780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:13:36.228547 4780 webserver.cc:460] Webserver started at http://127.17.31.62:33349/ using document root <none> and password file <none>
I20260807 03:13:36.229214 4780 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:36.229355 4780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:36.229704 4780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:36.232551 4780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "a846614fb1ca4ac1a873a8983260f52c"
format_stamp: "Formatted at 2026-08-07 03:13:36 on dist-test-slave-tm1g"
server_key: "faa8c9295081d73a2bbe020d15983cb3"
server_key_iv: "3744d60cf58cfbd16e2c20c484410b1f"
server_key_version: "encryptionkey@0"
I20260807 03:13:36.233409 4780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "a846614fb1ca4ac1a873a8983260f52c"
format_stamp: "Formatted at 2026-08-07 03:13:36 on dist-test-slave-tm1g"
server_key: "faa8c9295081d73a2bbe020d15983cb3"
server_key_iv: "3744d60cf58cfbd16e2c20c484410b1f"
server_key_version: "encryptionkey@0"
I20260807 03:13:36.238592 4780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 03:13:36.241791 4794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:36.243139 4780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:13:36.243358 4780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "a846614fb1ca4ac1a873a8983260f52c"
format_stamp: "Formatted at 2026-08-07 03:13:36 on dist-test-slave-tm1g"
server_key: "faa8c9295081d73a2bbe020d15983cb3"
server_key_iv: "3744d60cf58cfbd16e2c20c484410b1f"
server_key_version: "encryptionkey@0"
I20260807 03:13:36.243577 4780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:36.256757 4780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:36.257472 4780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:36.257752 4780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:36.275892 4780 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:37745
I20260807 03:13:36.275923 4848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:37745 every 8 connection(s)
I20260807 03:13:36.277149 4780 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:13:36.280766 4849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:36.285772 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 4780
I20260807 03:13:36.285959 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:13:36.286521 17532 external_mini_cluster.cc:1428] Setting key d082e3037aabfd10019428273fb21699
I20260807 03:13:36.292181 4849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c: Bootstrap starting.
I20260807 03:13:36.295567 4849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:36.296659 4849 log.cc:824] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:36.299726 4849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c: No bootstrap required, opened a new log
I20260807 03:13:36.300144 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:36.288383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.17.31.62:37745)
0807 03:13:36.289397 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:36.289410 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:36.289451 (+ 41us) server_negotiation.cc:407] Connection header received
0807 03:13:36.291136 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:36.291157 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:36.291554 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:36.291848 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:36.292502 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:36.293767 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:36.295722 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:36.296397 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:36.296524 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:36.297861 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:36.297872 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:36.297891 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:36.298512 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:36.298585 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:36.298610 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:36.298694 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:36.298933 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 03:13:36.299241 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":417,"threads_started":1}
I20260807 03:13:36.306885 4849 raft_consensus.cc:348] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a846614fb1ca4ac1a873a8983260f52c" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37745 } }
I20260807 03:13:36.307222 4849 raft_consensus.cc:374] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:36.307360 4849 raft_consensus.cc:729] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a846614fb1ca4ac1a873a8983260f52c, State: Initialized, Role: FOLLOWER
I20260807 03:13:36.307973 4849 consensus_queue.cc:260] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a846614fb1ca4ac1a873a8983260f52c" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37745 } }
I20260807 03:13:36.308213 4849 raft_consensus.cc:388] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:13:36.308359 4849 raft_consensus.cc:482] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:13:36.308554 4849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:36.310679 4849 raft_consensus.cc:504] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a846614fb1ca4ac1a873a8983260f52c" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37745 } }
I20260807 03:13:36.311134 4849 leader_election.cc:302] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a846614fb1ca4ac1a873a8983260f52c; no voters:
I20260807 03:13:36.311637 4849 leader_election.cc:288] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:13:36.311765 4855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:36.312474 4855 raft_consensus.cc:686] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [term 1 LEADER]: Becoming Leader. State: Replica: a846614fb1ca4ac1a873a8983260f52c, State: Running, Role: LEADER
I20260807 03:13:36.312960 4855 consensus_queue.cc:237] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a846614fb1ca4ac1a873a8983260f52c" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37745 } }
I20260807 03:13:36.313591 4849 sys_catalog.cc:564] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:13:36.315748 4857 sys_catalog.cc:455] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a846614fb1ca4ac1a873a8983260f52c. Latest consensus state: current_term: 1 leader_uuid: "a846614fb1ca4ac1a873a8983260f52c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a846614fb1ca4ac1a873a8983260f52c" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37745 } } }
I20260807 03:13:36.316248 4857 sys_catalog.cc:458] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:36.315876 4856 sys_catalog.cc:455] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a846614fb1ca4ac1a873a8983260f52c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a846614fb1ca4ac1a873a8983260f52c" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37745 } } }
I20260807 03:13:36.316722 4856 sys_catalog.cc:458] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:36.318111 4864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:13:36.323089 4864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:13:36.331527 4864 catalog_manager.cc:1347] Generated new cluster ID: 45c972822ec64ae4be4a57597c469172
I20260807 03:13:36.331645 4864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:13:36.352270 4864 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:13:36.353488 4864 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:13:36.364888 4864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c: Generated new TSK 0
I20260807 03:13:36.365888 4864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:13:36.379940 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37745
--builtin_ntp_servers=127.17.31.20:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 03:13:36.714398 4875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:36.714700 4875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:36.714854 4875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:36.722229 4875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:36.722373 4875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:36.722546 4875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:13:36.731593 4875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:36.732875 4875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:36.734630 4875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:36.747568 4880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:36.747535 4881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:36.747730 4883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:36.748653 4875 server_base.cc:1034] running on GCE node
I20260807 03:13:36.749292 4875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:36.750204 4875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:36.751425 4875 hybrid_clock.cc:648] HybridClock initialized: now 1786072416751352 us; error 60 us; skew 500 ppm
I20260807 03:13:36.751732 4875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:13:36.753842 4875 webserver.cc:460] Webserver started at http://127.17.31.1:35585/ using document root <none> and password file <none>
I20260807 03:13:36.754616 4875 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:36.754748 4875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:36.755069 4875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:36.757869 4875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "454d44bf1ebc4ccda559a18ddc3567f6"
format_stamp: "Formatted at 2026-08-07 03:13:36 on dist-test-slave-tm1g"
server_key: "f56409d29e97cbc4d82303550e9bfa37"
server_key_iv: "966d20007bfebfe5d0974180cdb51cdd"
server_key_version: "encryptionkey@0"
I20260807 03:13:36.758701 4875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "454d44bf1ebc4ccda559a18ddc3567f6"
format_stamp: "Formatted at 2026-08-07 03:13:36 on dist-test-slave-tm1g"
server_key: "f56409d29e97cbc4d82303550e9bfa37"
server_key_iv: "966d20007bfebfe5d0974180cdb51cdd"
server_key_version: "encryptionkey@0"
I20260807 03:13:36.763734 4875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260807 03:13:36.767134 4889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:36.768338 4875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:13:36.768515 4875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "454d44bf1ebc4ccda559a18ddc3567f6"
format_stamp: "Formatted at 2026-08-07 03:13:36 on dist-test-slave-tm1g"
server_key: "f56409d29e97cbc4d82303550e9bfa37"
server_key_iv: "966d20007bfebfe5d0974180cdb51cdd"
server_key_version: "encryptionkey@0"
I20260807 03:13:36.768723 4875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:36.785959 4875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:36.786774 4875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:36.787006 4875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:36.787882 4875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:36.789196 4875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:36.789283 4875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:36.789354 4875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:36.789410 4875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:36.822754 4875 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:38425
I20260807 03:13:36.822829 5006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:38425 every 8 connection(s)
I20260807 03:13:36.824103 4875 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:13:36.827121 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 4875
I20260807 03:13:36.827319 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:13:36.827782 17532 external_mini_cluster.cc:1428] Setting key df4e23f8b4bde1eef209297f24b1d01d
I20260807 03:13:36.830912 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37745
--builtin_ntp_servers=127.17.31.20:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 03:13:36.838027 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:36.826675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:52123 (local address 127.17.31.62:37745)
0807 03:13:36.827156 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:36.827162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:36.828609 (+ 1447us) server_negotiation.cc:407] Connection header received
0807 03:13:36.830681 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:36.830688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:36.830793 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:36.830921 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:36.832005 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:36.833298 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:36.834915 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:36.835632 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:36.835770 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:36.837062 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:36.837070 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:36.837078 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:36.837101 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:36.837156 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:36.837177 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:36.837274 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:36.837467 (+ 193us) server_negotiation.cc:299] Negotiation successful
0807 03:13:36.837578 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":253,"threads_started":1}
I20260807 03:13:36.838760 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:36.827351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:37745 (local address 127.17.31.1:52123)
0807 03:13:36.828455 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:36.828500 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:36.830422 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:36.831210 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:36.831221 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:36.831311 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:36.831843 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:36.831858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:36.833437 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:36.833441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:36.834771 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:36.834783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:36.835845 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:36.835849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:36.836051 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:36.836762 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:13:36.836764 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:36.836802 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:13:36.836878 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:36.837275 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:36.837282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:36.837287 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 03:13:36.837419 (+ 132us) client_negotiation.cc:240] Negotiation successful
0807 03:13:36.837693 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":366,"threads_started":1}
I20260807 03:13:36.840220 5007 heartbeater.cc:347] Connected to a master server at 127.17.31.62:37745
I20260807 03:13:36.840520 5007 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:36.841398 5007 heartbeater.cc:511] Master 127.17.31.62:37745 requested a full tablet report, sending...
I20260807 03:13:36.843772 4812 ts_manager.cc:194] Registered new tserver with Master: 454d44bf1ebc4ccda559a18ddc3567f6 (127.17.31.1:38425)
I20260807 03:13:36.846058 4812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.31.1:52123
W20260807 03:13:37.169951 5011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:37.170236 5011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:37.170409 5011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:37.177500 5011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:37.177585 5011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:37.177721 5011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:13:37.186604 5011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:37.187862 5011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:37.189587 5011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:37.202677 5016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:37.202631 5019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:37.202724 5017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:37.203456 5011 server_base.cc:1034] running on GCE node
I20260807 03:13:37.204016 5011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:37.204833 5011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:37.206027 5011 hybrid_clock.cc:648] HybridClock initialized: now 1786072417205978 us; error 49 us; skew 500 ppm
I20260807 03:13:37.206405 5011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:13:37.208622 5011 webserver.cc:460] Webserver started at http://127.17.31.2:37505/ using document root <none> and password file <none>
I20260807 03:13:37.209286 5011 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:37.209406 5011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:37.209734 5011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:37.212581 5011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "87359a500ed1437490b0b92499bf49d8"
format_stamp: "Formatted at 2026-08-07 03:13:37 on dist-test-slave-tm1g"
server_key: "7d4e6a4714d75a6c14df2a69b2e8e838"
server_key_iv: "1e74f8c45c9f0867eb9f6cff9a9ea127"
server_key_version: "encryptionkey@0"
I20260807 03:13:37.213385 5011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "87359a500ed1437490b0b92499bf49d8"
format_stamp: "Formatted at 2026-08-07 03:13:37 on dist-test-slave-tm1g"
server_key: "7d4e6a4714d75a6c14df2a69b2e8e838"
server_key_iv: "1e74f8c45c9f0867eb9f6cff9a9ea127"
server_key_version: "encryptionkey@0"
I20260807 03:13:37.218412 5011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 03:13:37.221871 5025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:37.223140 5011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:13:37.223315 5011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "87359a500ed1437490b0b92499bf49d8"
format_stamp: "Formatted at 2026-08-07 03:13:37 on dist-test-slave-tm1g"
server_key: "7d4e6a4714d75a6c14df2a69b2e8e838"
server_key_iv: "1e74f8c45c9f0867eb9f6cff9a9ea127"
server_key_version: "encryptionkey@0"
I20260807 03:13:37.223528 5011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:37.239249 5011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:37.239957 5011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:37.240217 5011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:37.241093 5011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:37.242447 5011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:37.242568 5011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:37.242689 5011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:37.242774 5011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:37.285356 5011 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:45309
I20260807 03:13:37.285375 5144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:45309 every 8 connection(s)
I20260807 03:13:37.286849 5011 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:13:37.291916 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 5011
I20260807 03:13:37.292162 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:13:37.292873 17532 external_mini_cluster.cc:1428] Setting key 5764406d3efd70463ef5004398c2c212
I20260807 03:13:37.297068 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37745
--builtin_ntp_servers=127.17.31.20:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 03:13:37.315122 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.290550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:51391 (local address 127.17.31.62:37745)
0807 03:13:37.290800 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:37.290810 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:37.303383 (+ 12573us) server_negotiation.cc:407] Connection header received
0807 03:13:37.305530 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:37.305539 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:37.305632 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:37.305749 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:37.307121 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.308500 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.311326 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.312058 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.312195 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.314026 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:37.314035 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:37.314043 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:37.314068 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:37.314111 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:37.314129 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:37.314229 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:37.314622 (+ 393us) server_negotiation.cc:299] Negotiation successful
0807 03:13:37.314735 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 03:13:37.316552 5147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.290150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:37745 (local address 127.17.31.2:51391)
0807 03:13:37.303198 (+ 13048us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:37.303250 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:37.305250 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:37.306046 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:37.306058 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:37.306168 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:37.306938 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.306959 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.308726 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.308735 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:37.311104 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.311123 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.312302 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.312309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:37.312675 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.313602 (+ 927us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:13:37.313606 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:37.313660 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:13:37.313786 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:37.314245 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:37.314260 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:37.314269 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 03:13:37.314585 (+ 316us) client_negotiation.cc:240] Negotiation successful
0807 03:13:37.315058 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12462,"thread_start_us":11917,"threads_started":1}
I20260807 03:13:37.318441 5145 heartbeater.cc:347] Connected to a master server at 127.17.31.62:37745
I20260807 03:13:37.318738 5145 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:37.319630 5145 heartbeater.cc:511] Master 127.17.31.62:37745 requested a full tablet report, sending...
I20260807 03:13:37.320976 4812 ts_manager.cc:194] Registered new tserver with Master: 87359a500ed1437490b0b92499bf49d8 (127.17.31.2:45309)
I20260807 03:13:37.321789 4812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.31.2:51391
W20260807 03:13:37.641863 5148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:37.642156 5148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:37.642335 5148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:37.650061 5148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:37.650188 5148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:37.650398 5148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:13:37.659195 5148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:37.660473 5148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:37.662222 5148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:37.674774 5153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:37.674866 5154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:37.674989 5156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:37.675777 5148 server_base.cc:1034] running on GCE node
I20260807 03:13:37.676404 5148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:37.677264 5148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:37.678501 5148 hybrid_clock.cc:648] HybridClock initialized: now 1786072417678451 us; error 41 us; skew 500 ppm
I20260807 03:13:37.678798 5148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:13:37.680878 5148 webserver.cc:460] Webserver started at http://127.17.31.3:36203/ using document root <none> and password file <none>
I20260807 03:13:37.681563 5148 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:37.681692 5148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:37.682003 5148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:37.684718 5148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "85ff037a366048b8bc87c0859688153e"
format_stamp: "Formatted at 2026-08-07 03:13:37 on dist-test-slave-tm1g"
server_key: "0348efa2c4a97337727a17130d4c82a6"
server_key_iv: "5a4f17d2b0e323ce30121537a874dd13"
server_key_version: "encryptionkey@0"
I20260807 03:13:37.685506 5148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "85ff037a366048b8bc87c0859688153e"
format_stamp: "Formatted at 2026-08-07 03:13:37 on dist-test-slave-tm1g"
server_key: "0348efa2c4a97337727a17130d4c82a6"
server_key_iv: "5a4f17d2b0e323ce30121537a874dd13"
server_key_version: "encryptionkey@0"
I20260807 03:13:37.690425 5148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260807 03:13:37.693605 5162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:37.694854 5148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260807 03:13:37.695050 5148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "85ff037a366048b8bc87c0859688153e"
format_stamp: "Formatted at 2026-08-07 03:13:37 on dist-test-slave-tm1g"
server_key: "0348efa2c4a97337727a17130d4c82a6"
server_key_iv: "5a4f17d2b0e323ce30121537a874dd13"
server_key_version: "encryptionkey@0"
I20260807 03:13:37.695255 5148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:37.715447 5148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:37.716182 5148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:37.716477 5148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:37.717386 5148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:37.718923 5148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:37.719046 5148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:37.719164 5148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:37.719252 5148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:37.760495 5148 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:35967
I20260807 03:13:37.760581 5279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:35967 every 8 connection(s)
I20260807 03:13:37.761773 5148 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:13:37.769409 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 5148
I20260807 03:13:37.769591 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:13:37.770041 17532 external_mini_cluster.cc:1428] Setting key 2962c588ee83591d58503d392766a88c
I20260807 03:13:37.774749 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.764366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:37031 (local address 127.17.31.62:37745)
0807 03:13:37.764553 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:37.764557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:37.765930 (+ 1373us) server_negotiation.cc:407] Connection header received
0807 03:13:37.767812 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:37.767819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:37.767911 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:37.768023 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:37.769035 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.769923 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.771829 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.772421 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.772500 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.774090 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:37.774094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:37.774098 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:37.774114 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:37.774147 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:37.774158 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:37.774230 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:37.774449 (+ 219us) server_negotiation.cc:299] Negotiation successful
0807 03:13:37.774525 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 03:13:37.775679 5282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.764889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:37745 (local address 127.17.31.3:37031)
0807 03:13:37.765765 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:37.765806 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:37.767537 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:37.768243 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:37.768251 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:37.768331 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:37.768871 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.768887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.770086 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.770090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:37.771594 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.771630 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.772878 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.772885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:37.773084 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.773806 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:13:37.773809 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:37.773849 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:13:37.773923 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:37.774244 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:37.774251 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:37.774257 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 03:13:37.774431 (+ 174us) client_negotiation.cc:240] Negotiation successful
0807 03:13:37.774693 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":271,"threads_started":1}
I20260807 03:13:37.777093 5280 heartbeater.cc:347] Connected to a master server at 127.17.31.62:37745
I20260807 03:13:37.777369 5280 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:37.778087 5280 heartbeater.cc:511] Master 127.17.31.62:37745 requested a full tablet report, sending...
I20260807 03:13:37.779593 4812 ts_manager.cc:194] Registered new tserver with Master: 85ff037a366048b8bc87c0859688153e (127.17.31.3:35967)
I20260807 03:13:37.780401 4812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.31.3:37031
I20260807 03:13:37.784835 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:13:37.794646 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.788845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.17.31.62:37745)
0807 03:13:37.788995 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:37.789000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:37.789504 (+ 504us) server_negotiation.cc:407] Connection header received
0807 03:13:37.789627 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:37.789631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:37.789698 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:37.789777 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:37.790233 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.790940 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.792452 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.792877 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.792979 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.793964 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:37.793969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:37.793972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:37.793989 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:37.794018 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:37.794029 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:37.794130 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:37.794285 (+ 155us) server_negotiation.cc:299] Negotiation successful
0807 03:13:37.794437 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 03:13:37.798224 4812 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:13:37.805020 4812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:13:37.831841 5294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.816514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.17.31.1:38425)
0807 03:13:37.817102 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:37.817110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:37.819839 (+ 2729us) server_negotiation.cc:407] Connection header received
0807 03:13:37.820147 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:37.820159 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:37.820456 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:37.820675 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:37.821898 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.824655 (+ 2757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.827943 (+ 3288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.829798 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.829944 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.830293 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:37.831160 (+ 867us) server_negotiation.cc:299] Negotiation successful
0807 03:13:37.831392 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":317,"threads_started":1}
I20260807 03:13:37.832778 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.816277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:38425 (local address 127.0.0.1:47312)
0807 03:13:37.819650 (+ 3373us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:37.819690 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:37.819893 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:37.821021 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:37.821033 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:37.821076 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:37.821662 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.821678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.824819 (+ 3141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.824824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:37.827689 (+ 2865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.827702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.830519 (+ 2817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.830528 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:37.830816 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.830884 (+ 68us) client_negotiation.cc:769] Sending connection context
0807 03:13:37.832190 (+ 1306us) client_negotiation.cc:240] Negotiation successful
0807 03:13:37.832375 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3101,"mutex_wait_us":27,"thread_start_us":2918,"threads_started":1}
I20260807 03:13:37.836994 4940 tablet_service.cc:1467] Processing CreateTablet for tablet 7fba34a7dfbc450196a9cf3824db2300 (DEFAULT_TABLE table=test-table [id=9394dd5d8ca840d5a0661871711102e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:37.838496 4940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fba34a7dfbc450196a9cf3824db2300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:37.840530 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.817202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:45309 (local address 127.0.0.1:57886)
0807 03:13:37.820825 (+ 3623us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:37.820840 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:37.820945 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:37.833969 (+ 13024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:37.833978 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:37.834014 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:37.834446 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.834458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.836048 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.836055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:37.837945 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.837961 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.839788 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.839795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:37.839961 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.839997 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 03:13:37.840093 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 03:13:37.840224 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3506,"thread_start_us":1310,"threads_started":1}
I20260807 03:13:37.840672 5297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.817957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57886 (local address 127.17.31.2:45309)
0807 03:13:37.833369 (+ 15412us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:37.833378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:37.833422 (+ 44us) server_negotiation.cc:407] Connection header received
0807 03:13:37.833529 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:37.833537 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:37.833753 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:37.833947 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:37.834637 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.835921 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.838156 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.839587 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.839735 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.839949 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:37.840159 (+ 210us) server_negotiation.cc:299] Negotiation successful
0807 03:13:37.840313 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15195,"thread_start_us":16147,"threads_started":1}
I20260807 03:13:37.841863 5295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.816479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:35967 (local address 127.0.0.1:58774)
0807 03:13:37.820347 (+ 3868us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:37.820366 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:37.820473 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:37.835350 (+ 14877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:37.835356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:37.835384 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:37.835718 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.835727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.837616 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.837621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:37.839177 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.839189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.841103 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.841110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:37.841294 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.841326 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 03:13:37.841422 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 03:13:37.841556 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3699,"thread_start_us":4113,"threads_started":1}
I20260807 03:13:37.842721 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.817786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.17.31.3:35967)
0807 03:13:37.834789 (+ 17003us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:37.834797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:37.834830 (+ 33us) server_negotiation.cc:407] Connection header received
0807 03:13:37.834917 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:37.834925 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:37.835142 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:37.835409 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:37.835902 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.837421 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.839365 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.840704 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.840852 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.841111 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:37.841496 (+ 385us) server_negotiation.cc:299] Negotiation successful
0807 03:13:37.841654 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16806,"thread_start_us":16638,"threads_started":1}
I20260807 03:13:37.844195 5077 tablet_service.cc:1467] Processing CreateTablet for tablet 7fba34a7dfbc450196a9cf3824db2300 (DEFAULT_TABLE table=test-table [id=9394dd5d8ca840d5a0661871711102e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:37.845271 5077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fba34a7dfbc450196a9cf3824db2300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:37.849123 5007 heartbeater.cc:503] Master 127.17.31.62:37745 was elected leader, sending a full tablet report...
I20260807 03:13:37.848769 5213 tablet_service.cc:1467] Processing CreateTablet for tablet 7fba34a7dfbc450196a9cf3824db2300 (DEFAULT_TABLE table=test-table [id=9394dd5d8ca840d5a0661871711102e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:37.850170 5213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fba34a7dfbc450196a9cf3824db2300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:37.855089 5298 tablet_bootstrap.cc:492] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6: Bootstrap starting.
I20260807 03:13:37.858031 5298 tablet_bootstrap.cc:654] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:37.859365 5298 log.cc:824] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:37.865517 5299 tablet_bootstrap.cc:492] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8: Bootstrap starting.
I20260807 03:13:37.868669 5301 tablet_bootstrap.cc:492] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e: Bootstrap starting.
I20260807 03:13:37.869272 5298 tablet_bootstrap.cc:492] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6: No bootstrap required, opened a new log
I20260807 03:13:37.869385 5299 tablet_bootstrap.cc:654] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:37.869554 5298 ts_tablet_manager.cc:1397] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260807 03:13:37.870645 5299 log.cc:824] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:37.872540 5301 tablet_bootstrap.cc:654] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:37.873363 5299 tablet_bootstrap.cc:492] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8: No bootstrap required, opened a new log
I20260807 03:13:37.873654 5299 ts_tablet_manager.cc:1397] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260807 03:13:37.873693 5301 log.cc:824] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:37.876639 5301 tablet_bootstrap.cc:492] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e: No bootstrap required, opened a new log
I20260807 03:13:37.876879 5301 ts_tablet_manager.cc:1397] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260807 03:13:37.879961 5298 raft_consensus.cc:348] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38425 } } peers { permanent_uuid: "85ff037a366048b8bc87c0859688153e" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35967 } } peers { permanent_uuid: "87359a500ed1437490b0b92499bf49d8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 45309 } }
I20260807 03:13:37.880401 5298 raft_consensus.cc:374] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:37.880560 5298 raft_consensus.cc:729] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 454d44bf1ebc4ccda559a18ddc3567f6, State: Initialized, Role: FOLLOWER
I20260807 03:13:37.880895 5299 raft_consensus.cc:348] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38425 } } peers { permanent_uuid: "85ff037a366048b8bc87c0859688153e" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35967 } } peers { permanent_uuid: "87359a500ed1437490b0b92499bf49d8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 45309 } }
I20260807 03:13:37.881342 5299 raft_consensus.cc:374] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:37.881194 5298 consensus_queue.cc:260] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38425 } } peers { permanent_uuid: "85ff037a366048b8bc87c0859688153e" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35967 } } peers { permanent_uuid: "87359a500ed1437490b0b92499bf49d8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 45309 } }
I20260807 03:13:37.881441 5299 raft_consensus.cc:729] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87359a500ed1437490b0b92499bf49d8, State: Initialized, Role: FOLLOWER
I20260807 03:13:37.882154 5299 consensus_queue.cc:260] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38425 } } peers { permanent_uuid: "85ff037a366048b8bc87c0859688153e" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35967 } } peers { permanent_uuid: "87359a500ed1437490b0b92499bf49d8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 45309 } }
I20260807 03:13:37.882869 5298 ts_tablet_manager.cc:1428] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260807 03:13:37.884989 5145 heartbeater.cc:503] Master 127.17.31.62:37745 was elected leader, sending a full tablet report...
I20260807 03:13:37.886466 5301 raft_consensus.cc:348] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38425 } } peers { permanent_uuid: "85ff037a366048b8bc87c0859688153e" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35967 } } peers { permanent_uuid: "87359a500ed1437490b0b92499bf49d8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 45309 } }
I20260807 03:13:37.886940 5299 ts_tablet_manager.cc:1428] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260807 03:13:37.886970 5301 raft_consensus.cc:374] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:37.887099 5301 raft_consensus.cc:729] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ff037a366048b8bc87c0859688153e, State: Initialized, Role: FOLLOWER
I20260807 03:13:37.887822 5301 consensus_queue.cc:260] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38425 } } peers { permanent_uuid: "85ff037a366048b8bc87c0859688153e" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35967 } } peers { permanent_uuid: "87359a500ed1437490b0b92499bf49d8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 45309 } }
I20260807 03:13:37.889312 5280 heartbeater.cc:503] Master 127.17.31.62:37745 was elected leader, sending a full tablet report...
I20260807 03:13:37.889911 5301 ts_tablet_manager.cc:1428] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260807 03:13:37.975703 5304 raft_consensus.cc:482] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:13:37.975994 5304 raft_consensus.cc:504] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38425 } } peers { permanent_uuid: "85ff037a366048b8bc87c0859688153e" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35967 } } peers { permanent_uuid: "87359a500ed1437490b0b92499bf49d8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 45309 } }
I20260807 03:13:37.977672 5304 leader_election.cc:288] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85ff037a366048b8bc87c0859688153e (127.17.31.3:35967), 87359a500ed1437490b0b92499bf49d8 (127.17.31.2:45309)
I20260807 03:13:37.979679 5305 raft_consensus.cc:482] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:13:37.980063 5305 raft_consensus.cc:504] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38425 } } peers { permanent_uuid: "85ff037a366048b8bc87c0859688153e" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35967 } } peers { permanent_uuid: "87359a500ed1437490b0b92499bf49d8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 45309 } }
I20260807 03:13:37.981657 5305 leader_election.cc:288] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 454d44bf1ebc4ccda559a18ddc3567f6 (127.17.31.1:38425), 85ff037a366048b8bc87c0859688153e (127.17.31.3:35967)
I20260807 03:13:37.986531 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.978068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:54285 (local address 127.17.31.3:35967)
0807 03:13:37.978275 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:37.978283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:37.978860 (+ 577us) server_negotiation.cc:407] Connection header received
0807 03:13:37.979099 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:37.979105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:37.979177 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:37.979276 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:37.979930 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.980871 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.983726 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.985361 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.985444 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.985558 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:37.986108 (+ 550us) server_negotiation.cc:299] Negotiation successful
0807 03:13:37.986236 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 03:13:37.986608 5297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.978223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:49633 (local address 127.17.31.2:45309)
0807 03:13:37.978493 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:37.978501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:37.979305 (+ 804us) server_negotiation.cc:407] Connection header received
0807 03:13:37.979543 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:37.979550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:37.979655 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:37.979759 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:37.980439 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.981474 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.984000 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.985230 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.985358 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.985474 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:37.986120 (+ 646us) server_negotiation.cc:299] Negotiation successful
0807 03:13:37.986240 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 03:13:37.987381 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.977928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:45309 (local address 127.17.31.1:49633)
0807 03:13:37.979200 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:37.979215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:37.979387 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:37.979796 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:37.979802 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:37.979834 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:37.980258 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.980274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.981647 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.981652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:37.983785 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.983799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.985651 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.985657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:37.985850 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.985948 (+ 98us) client_negotiation.cc:769] Sending connection context
0807 03:13:37.986809 (+ 861us) client_negotiation.cc:240] Negotiation successful
0807 03:13:37.986972 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"mutex_wait_us":109,"thread_start_us":470,"threads_started":1}
I20260807 03:13:37.987411 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.977928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:35967 (local address 127.17.31.1:54285)
0807 03:13:37.978748 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:37.978764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:37.978945 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:37.979303 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:37.979308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:37.979334 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:37.979745 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.979759 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.981043 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.981048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:37.983443 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.983458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.985707 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.985712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:37.985918 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.985964 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 03:13:37.986957 (+ 993us) client_negotiation.cc:240] Negotiation successful
0807 03:13:37.987055 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":59,"thread_start_us":923,"threads_started":1}
I20260807 03:13:37.988781 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fba34a7dfbc450196a9cf3824db2300" candidate_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ff037a366048b8bc87c0859688153e" is_pre_election: true
I20260807 03:13:37.989276 5233 raft_consensus.cc:2454] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 454d44bf1ebc4ccda559a18ddc3567f6 in term 0.
I20260807 03:13:37.989913 4890 leader_election.cc:302] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 454d44bf1ebc4ccda559a18ddc3567f6, 85ff037a366048b8bc87c0859688153e; no voters:
I20260807 03:13:37.990281 5304 raft_consensus.cc:2781] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:13:37.990543 5304 raft_consensus.cc:482] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:13:37.990710 5304 raft_consensus.cc:3037] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:37.991487 5098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fba34a7dfbc450196a9cf3824db2300" candidate_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87359a500ed1437490b0b92499bf49d8" is_pre_election: true
I20260807 03:13:37.992008 5098 raft_consensus.cc:2454] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 454d44bf1ebc4ccda559a18ddc3567f6 in term 0.
I20260807 03:13:37.995972 5304 raft_consensus.cc:504] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38425 } } peers { permanent_uuid: "85ff037a366048b8bc87c0859688153e" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35967 } } peers { permanent_uuid: "87359a500ed1437490b0b92499bf49d8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 45309 } }
I20260807 03:13:37.997115 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fba34a7dfbc450196a9cf3824db2300" candidate_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ff037a366048b8bc87c0859688153e"
I20260807 03:13:37.997335 5233 raft_consensus.cc:3037] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:37.998203 5294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.982483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:33861 (local address 127.17.31.1:38425)
0807 03:13:37.982857 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:37.982864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:37.990941 (+ 8077us) server_negotiation.cc:407] Connection header received
0807 03:13:37.991217 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:37.991223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:37.991306 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:37.991375 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:37.992276 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.993243 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.995828 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.997037 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.997145 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.997233 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:37.997832 (+ 599us) server_negotiation.cc:299] Negotiation successful
0807 03:13:37.997963 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260807 03:13:37.998839 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.981960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:38425 (local address 127.17.31.2:33861)
0807 03:13:37.990846 (+ 8886us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:37.990867 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:37.991048 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:37.991577 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:37.991584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:37.991621 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:37.992088 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.992106 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.993415 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.993421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:37.995594 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.995609 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.997415 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.997421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:37.997598 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:37.997640 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 03:13:37.998381 (+ 741us) client_negotiation.cc:240] Negotiation successful
0807 03:13:37.998522 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8708,"thread_start_us":14836,"threads_started":1}
I20260807 03:13:37.999814 4961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fba34a7dfbc450196a9cf3824db2300" candidate_uuid: "87359a500ed1437490b0b92499bf49d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" is_pre_election: true
I20260807 03:13:38.000260 4961 raft_consensus.cc:2379] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 87359a500ed1437490b0b92499bf49d8 in current term 1: Already voted for candidate 454d44bf1ebc4ccda559a18ddc3567f6 in this term.
I20260807 03:13:38.000833 5233 raft_consensus.cc:2454] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 454d44bf1ebc4ccda559a18ddc3567f6 in term 1.
I20260807 03:13:38.001237 5304 leader_election.cc:288] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [CANDIDATE]: Term 1 election: Requested vote from peers 85ff037a366048b8bc87c0859688153e (127.17.31.3:35967), 87359a500ed1437490b0b92499bf49d8 (127.17.31.2:45309)
I20260807 03:13:38.001284 4890 leader_election.cc:302] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 454d44bf1ebc4ccda559a18ddc3567f6, 85ff037a366048b8bc87c0859688153e; no voters:
I20260807 03:13:38.001582 5304 raft_consensus.cc:2781] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:38.001956 5098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fba34a7dfbc450196a9cf3824db2300" candidate_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87359a500ed1437490b0b92499bf49d8"
I20260807 03:13:38.002102 5098 raft_consensus.cc:3037] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:38.004288 5310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.982138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:35967 (local address 127.17.31.2:36059)
0807 03:13:37.997075 (+ 14937us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:37.997089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:37.997226 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:37.997574 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:37.997578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:37.997929 (+ 351us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:37.998270 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:37.998288 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.999752 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:37.999755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:38.001443 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:38.001457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:38.003545 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:38.003552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:38.003723 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:38.003764 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 03:13:38.003863 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 03:13:38.003994 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14833,"thread_start_us":8328,"threads_started":1}
I20260807 03:13:38.004678 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:37.982479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:36059 (local address 127.17.31.3:35967)
0807 03:13:37.986810 (+ 4331us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:37.986815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:37.997173 (+ 10358us) server_negotiation.cc:407] Connection header received
0807 03:13:37.997374 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:37.997379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:37.997470 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:37.997560 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:37.998493 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:37.999593 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:38.001640 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:38.003338 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:38.003506 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:38.003641 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:38.003910 (+ 269us) server_negotiation.cc:299] Negotiation successful
0807 03:13:38.004087 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4205,"thread_start_us":14078,"threads_started":1}
I20260807 03:13:38.004770 5098 raft_consensus.cc:2454] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 454d44bf1ebc4ccda559a18ddc3567f6 in term 1.
I20260807 03:13:38.005390 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fba34a7dfbc450196a9cf3824db2300" candidate_uuid: "87359a500ed1437490b0b92499bf49d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ff037a366048b8bc87c0859688153e" is_pre_election: true
I20260807 03:13:38.005669 5233 raft_consensus.cc:2379] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 87359a500ed1437490b0b92499bf49d8 in current term 1: Already voted for candidate 454d44bf1ebc4ccda559a18ddc3567f6 in this term.
I20260807 03:13:38.006070 5304 raft_consensus.cc:686] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [term 1 LEADER]: Becoming Leader. State: Replica: 454d44bf1ebc4ccda559a18ddc3567f6, State: Running, Role: LEADER
I20260807 03:13:38.006274 5026 leader_election.cc:302] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87359a500ed1437490b0b92499bf49d8; no voters: 454d44bf1ebc4ccda559a18ddc3567f6, 85ff037a366048b8bc87c0859688153e
I20260807 03:13:38.006712 5305 raft_consensus.cc:2726] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 03:13:38.006611 5304 consensus_queue.cc:237] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38425 } } peers { permanent_uuid: "85ff037a366048b8bc87c0859688153e" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35967 } } peers { permanent_uuid: "87359a500ed1437490b0b92499bf49d8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 45309 } }
I20260807 03:13:38.011322 4811 catalog_manager.cc:5495] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 454d44bf1ebc4ccda559a18ddc3567f6 (127.17.31.1). New cstate: current_term: 1 leader_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "454d44bf1ebc4ccda559a18ddc3567f6" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85ff037a366048b8bc87c0859688153e" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87359a500ed1437490b0b92499bf49d8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 45309 } health_report { overall_health: UNKNOWN } } }
W20260807 03:13:38.013501 5281 tablet.cc:2369] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 03:13:38.054854 5146 tablet.cc:2369] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:13:38.067874 5294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:38.062237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47314 (local address 127.17.31.1:38425)
0807 03:13:38.062488 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:38.062496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:38.062569 (+ 73us) server_negotiation.cc:407] Connection header received
0807 03:13:38.062736 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:38.062740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:38.062851 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:38.062981 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:13:38.063298 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:38.064067 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:38.065412 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:38.065837 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:38.065940 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:38.066660 (+ 720us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:13:38.067151 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:13:38.067299 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:38.067476 (+ 177us) server_negotiation.cc:299] Negotiation successful
0807 03:13:38.067556 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260807 03:13:38.076278 5098 raft_consensus.cc:1261] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8 [term 1 FOLLOWER]: Refusing update from remote peer 454d44bf1ebc4ccda559a18ddc3567f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:13:38.076241 5233 raft_consensus.cc:1261] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e [term 1 FOLLOWER]: Refusing update from remote peer 454d44bf1ebc4ccda559a18ddc3567f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260807 03:13:38.076517 5008 tablet.cc:2369] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:13:38.077219 5313 consensus_queue.cc:1035] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ff037a366048b8bc87c0859688153e" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 35967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:13:38.077620 5304 consensus_queue.cc:1035] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87359a500ed1437490b0b92499bf49d8" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 45309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:13:38.091068 5316 mvcc.cc:205] Tried to move back new op lower bound from 7315752624430362624 to 7315752624159043584. Current Snapshot: MvccSnapshot[applied={T|T < 7315752624430362624}]
I20260807 03:13:38.094600 5318 mvcc.cc:205] Tried to move back new op lower bound from 7315752624430362624 to 7315752624159043584. Current Snapshot: MvccSnapshot[applied={T|T < 7315752624430362624 or (T in {7315752624430362624})}]
I20260807 03:13:38.105922 4811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:13:38.106211 4811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:13:38.109830 4811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a846614fb1ca4ac1a873a8983260f52c: Sending DeleteTablet for 3 replicas of tablet 7fba34a7dfbc450196a9cf3824db2300
I20260807 03:13:38.110877 4940 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fba34a7dfbc450196a9cf3824db2300 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:38 UTC) from {username='slave'} at 127.0.0.1:47312
I20260807 03:13:38.110980 5213 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fba34a7dfbc450196a9cf3824db2300 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:38 UTC) from {username='slave'} at 127.0.0.1:58774
I20260807 03:13:38.111122 5077 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fba34a7dfbc450196a9cf3824db2300 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:38 UTC) from {username='slave'} at 127.0.0.1:57886
I20260807 03:13:38.112866 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 4875
I20260807 03:13:38.124141 5326 tablet_replica.cc:330] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6: stopping tablet replica
I20260807 03:13:38.124799 5326 raft_consensus.cc:2229] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:13:38.125803 5326 raft_consensus.cc:2258] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:38.126748 5327 tablet_replica.cc:330] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e: stopping tablet replica
I20260807 03:13:38.127131 5327 raft_consensus.cc:2229] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:13:38.127523 5327 raft_consensus.cc:2258] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:38.129278 5326 ts_tablet_manager.cc:1905] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:38.134665 5326 ts_tablet_manager.cc:1918] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:38.134835 5326 log.cc:1196] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/7fba34a7dfbc450196a9cf3824db2300
I20260807 03:13:38.135387 5326 ts_tablet_manager.cc:1939] T 7fba34a7dfbc450196a9cf3824db2300 P 454d44bf1ebc4ccda559a18ddc3567f6: Deleting consensus metadata
I20260807 03:13:38.135695 5327 ts_tablet_manager.cc:1905] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:38.137194 4798 catalog_manager.cc:4841] TS 454d44bf1ebc4ccda559a18ddc3567f6 (127.17.31.1:38425): tablet 7fba34a7dfbc450196a9cf3824db2300 (table test-table [id=9394dd5d8ca840d5a0661871711102e9]) successfully deleted
I20260807 03:13:38.138113 5328 tablet_replica.cc:330] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8: stopping tablet replica
I20260807 03:13:38.138567 5328 raft_consensus.cc:2229] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:13:38.139047 5328 raft_consensus.cc:2258] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:38.141017 5328 ts_tablet_manager.cc:1905] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:38.143254 5327 ts_tablet_manager.cc:1918] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:38.143415 5327 log.cc:1196] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/7fba34a7dfbc450196a9cf3824db2300
I20260807 03:13:38.143950 5327 ts_tablet_manager.cc:1939] T 7fba34a7dfbc450196a9cf3824db2300 P 85ff037a366048b8bc87c0859688153e: Deleting consensus metadata
I20260807 03:13:38.145758 4795 catalog_manager.cc:4841] TS 85ff037a366048b8bc87c0859688153e (127.17.31.3:35967): tablet 7fba34a7dfbc450196a9cf3824db2300 (table test-table [id=9394dd5d8ca840d5a0661871711102e9]) successfully deleted
I20260807 03:13:38.146703 5294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:38.113993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47320 (local address 127.17.31.1:38425)
0807 03:13:38.114205 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:38.114213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:38.137945 (+ 23732us) server_negotiation.cc:407] Connection header received
0807 03:13:38.138353 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:38.138359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:38.138468 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:38.138553 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:38.139388 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:38.140553 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:38.142515 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:38.143161 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:38.143266 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:38.144843 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:38.144860 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:38.144876 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:38.145673 (+ 797us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:38.145730 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:38.145754 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:38.145842 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:38.146190 (+ 348us) server_negotiation.cc:299] Negotiation successful
0807 03:13:38.146324 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 03:13:38.147444 5002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:38.149339 5328 ts_tablet_manager.cc:1918] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:38.149533 5328 log.cc:1196] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/7fba34a7dfbc450196a9cf3824db2300
I20260807 03:13:38.150177 5328 ts_tablet_manager.cc:1939] T 7fba34a7dfbc450196a9cf3824db2300 P 87359a500ed1437490b0b92499bf49d8: Deleting consensus metadata
I20260807 03:13:38.151908 4796 catalog_manager.cc:4841] TS 87359a500ed1437490b0b92499bf49d8 (127.17.31.2:45309): tablet 7fba34a7dfbc450196a9cf3824db2300 (table test-table [id=9394dd5d8ca840d5a0661871711102e9]) successfully deleted
I20260807 03:13:38.242986 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 4875
I20260807 03:13:38.263401 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 5011
I20260807 03:13:38.270833 5297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:38.264231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.17.31.2:45309)
0807 03:13:38.264447 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:38.264453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:38.264473 (+ 20us) server_negotiation.cc:407] Connection header received
0807 03:13:38.264664 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:38.264670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:38.264770 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:38.264882 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:38.265350 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:38.266093 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:38.267476 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:38.267907 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:38.268047 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:38.269171 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:38.269180 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:38.269194 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:38.269781 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:38.269854 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:38.269874 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:38.270000 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:38.270135 (+ 135us) server_negotiation.cc:299] Negotiation successful
0807 03:13:38.270242 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 03:13:38.271803 5140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:38.361478 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 5011
I20260807 03:13:38.381165 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 5148
I20260807 03:13:38.387589 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:38.381956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53630 (local address 127.17.31.3:35967)
0807 03:13:38.382138 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:38.382142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:38.382161 (+ 19us) server_negotiation.cc:407] Connection header received
0807 03:13:38.382391 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:38.382395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:38.382480 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:38.382565 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:38.382939 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:38.383666 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:38.384925 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:38.385336 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:38.385429 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:38.386410 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:38.386419 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:38.386434 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:38.387009 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:38.387054 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:38.387073 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:38.387159 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:38.387292 (+ 133us) server_negotiation.cc:299] Negotiation successful
0807 03:13:38.387375 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 03:13:38.388226 5274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:38.475406 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 5148
I20260807 03:13:38.493635 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 4780
I20260807 03:13:38.494839 4844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:38.603327 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 4780
2026-08-07T03:13:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2819 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:13:38 dist-test-slave-tm1g krb5kdc[5337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:13:38 dist-test-slave-tm1g krb5kdc[5337](info): set up 2 sockets
Aug 07 03:13:38 dist-test-slave-tm1g krb5kdc[5337](info): commencing operation
krb5kdc: starting...
W20260807 03:13:39.740867 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:13:39 dist-test-slave-tm1g krb5kdc[5337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:13:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:13:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:13:39.995633 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:35001
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:36033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:35001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:40.321367 5351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:40.321689 5351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:40.321856 5351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:40.329638 5351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:13:40.329788 5351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:40.329890 5351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:40.329994 5351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:13:40.330070 5351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:13:40.339946 5351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:36033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:35001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:35001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:40.341224 5351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:40.342646 5351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:40.354341 5356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:40.354363 5359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:40.354569 5357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:40.359817 5351 server_base.cc:1034] running on GCE node
I20260807 03:13:40.360606 5351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:40.361948 5351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:40.363152 5351 hybrid_clock.cc:648] HybridClock initialized: now 1786072420363116 us; error 31 us; skew 500 ppm
I20260807 03:13:40.363448 5351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:40 dist-test-slave-tm1g krb5kdc[5337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072420, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:40.371035 5351 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:13:40.372496 5351 webserver.cc:460] Webserver started at http://127.17.31.62:35425/ using document root <none> and password file <none>
I20260807 03:13:40.373193 5351 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:40.373298 5351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:40.373546 5351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:40.376587 5351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "8bed7e39392c41c8b6a86b5b35eafc9e"
format_stamp: "Formatted at 2026-08-07 03:13:40 on dist-test-slave-tm1g"
server_key: "3ae51bb1578d8ffb6452f909c0c8ac8d"
server_key_iv: "dd3dbb85f911fb4d4ce026bc30c9af0e"
server_key_version: "encryptionkey@0"
I20260807 03:13:40.377333 5351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "8bed7e39392c41c8b6a86b5b35eafc9e"
format_stamp: "Formatted at 2026-08-07 03:13:40 on dist-test-slave-tm1g"
server_key: "3ae51bb1578d8ffb6452f909c0c8ac8d"
server_key_iv: "dd3dbb85f911fb4d4ce026bc30c9af0e"
server_key_version: "encryptionkey@0"
I20260807 03:13:40.382402 5351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260807 03:13:40.385700 5366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:40.387079 5351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 03:13:40.387212 5351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "8bed7e39392c41c8b6a86b5b35eafc9e"
format_stamp: "Formatted at 2026-08-07 03:13:40 on dist-test-slave-tm1g"
server_key: "3ae51bb1578d8ffb6452f909c0c8ac8d"
server_key_iv: "dd3dbb85f911fb4d4ce026bc30c9af0e"
server_key_version: "encryptionkey@0"
I20260807 03:13:40.387408 5351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:40.406213 5351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:40.410461 5351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:40.410743 5351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:40.428265 5351 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:35001
I20260807 03:13:40.428298 5417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:35001 every 8 connection(s)
I20260807 03:13:40.429617 5351 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:13:40.433109 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:40.438453 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 5351
I20260807 03:13:40.438681 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:13:40.439157 17532 external_mini_cluster.cc:1428] Setting key 10cf319b7da7a5d14e78d323eae286a7
I20260807 03:13:40.443558 5418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e: Bootstrap starting.
I20260807 03:13:40.446727 5418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:40.447722 5418 log.cc:824] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:40.450273 5418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e: No bootstrap required, opened a new log
Aug 07 03:13:40 dist-test-slave-tm1g krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:13:40.457490 5418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bed7e39392c41c8b6a86b5b35eafc9e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35001 } }
I20260807 03:13:40.457850 5418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:40.457969 5418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bed7e39392c41c8b6a86b5b35eafc9e, State: Initialized, Role: FOLLOWER
I20260807 03:13:40.458598 5418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bed7e39392c41c8b6a86b5b35eafc9e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35001 } }
I20260807 03:13:40.458833 5418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:13:40.459009 5418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:13:40.459211 5418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:40.461122 5421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:40.440971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.17.31.62:35001)
0807 03:13:40.441828 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:40.441839 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:40.441881 (+ 42us) server_negotiation.cc:407] Connection header received
0807 03:13:40.443573 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:40.443594 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:40.444000 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:40.444386 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:40.445936 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:40.447365 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:40.448701 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:40.449364 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:40.449567 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:40.453650 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:40.453657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:40.453674 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:40.453719 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:40.456920 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:40.457617 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:40.457632 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:40.457637 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:40.457816 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:40.458362 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:40.458367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:40.458369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:40.458997 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:40.459283 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:40.459702 (+ 419us) server_negotiation.cc:299] Negotiation successful
0807 03:13:40.460081 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":426,"threads_started":1}
I20260807 03:13:40.461797 5418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bed7e39392c41c8b6a86b5b35eafc9e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35001 } }
I20260807 03:13:40.462261 5418 leader_election.cc:302] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bed7e39392c41c8b6a86b5b35eafc9e; no voters:
I20260807 03:13:40.462936 5418 leader_election.cc:288] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:13:40.463073 5423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:40.464285 5423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [term 1 LEADER]: Becoming Leader. State: Replica: 8bed7e39392c41c8b6a86b5b35eafc9e, State: Running, Role: LEADER
I20260807 03:13:40.464920 5418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:13:40.464859 5423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bed7e39392c41c8b6a86b5b35eafc9e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35001 } }
I20260807 03:13:40.471060 5427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bed7e39392c41c8b6a86b5b35eafc9e. Latest consensus state: current_term: 1 leader_uuid: "8bed7e39392c41c8b6a86b5b35eafc9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bed7e39392c41c8b6a86b5b35eafc9e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35001 } } }
I20260807 03:13:40.471388 5427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:40.471902 5425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bed7e39392c41c8b6a86b5b35eafc9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bed7e39392c41c8b6a86b5b35eafc9e" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 35001 } } }
I20260807 03:13:40.472224 5425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e [sys.catalog]: This master's current role is: LEADER
W20260807 03:13:40.475481 5440 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:13:40.475593 5440 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:13:40.475775 5441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:13:40.477320 5441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:13:40.484601 5441 catalog_manager.cc:1347] Generated new cluster ID: a7eb1bd4120148c6814725e2b7103b55
I20260807 03:13:40.484706 5441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:13:40.503111 5441 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:13:40.504900 5441 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:13:40.515015 5441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e: Generated new TSK 0
I20260807 03:13:40.515969 5441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.31.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
Entry for principal kudu/127.17.31.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab.
I20260807 03:13:40.578860 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35001
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:36033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:40.912575 5447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:40.912846 5447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:40.913051 5447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:40.920122 5447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:40.920231 5447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:40.920457 5447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:13:40.930017 5447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:36033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.1.keytab
--principal=kudu/127.17.31.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:40.931332 5447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:40.932880 5447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:40.946504 5453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:40.946471 5454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:40.946537 5456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:40.947275 5447 server_base.cc:1034] running on GCE node
I20260807 03:13:40.947948 5447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:40.948863 5447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:40.950146 5447 hybrid_clock.cc:648] HybridClock initialized: now 1786072420950088 us; error 52 us; skew 500 ppm
I20260807 03:13:40.950482 5447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:40 dist-test-slave-tm1g krb5kdc[5337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072420, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:40.953575 5447 init.cc:377] Logged in from keytab as kudu/127.17.31.1@KRBTEST.COM (short username kudu)
I20260807 03:13:40.955142 5447 webserver.cc:460] Webserver started at http://127.17.31.1:40239/ using document root <none> and password file <none>
I20260807 03:13:40.955826 5447 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:40.955978 5447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:40.956296 5447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:40.959193 5447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "6da241f9924c4120a7d0046941405c1d"
format_stamp: "Formatted at 2026-08-07 03:13:40 on dist-test-slave-tm1g"
server_key: "0ce17d418b2891ab857a9c9f799e60a1"
server_key_iv: "1aabbfc1e505b5e2973447ed8e15e3fc"
server_key_version: "encryptionkey@0"
I20260807 03:13:40.959977 5447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "6da241f9924c4120a7d0046941405c1d"
format_stamp: "Formatted at 2026-08-07 03:13:40 on dist-test-slave-tm1g"
server_key: "0ce17d418b2891ab857a9c9f799e60a1"
server_key_iv: "1aabbfc1e505b5e2973447ed8e15e3fc"
server_key_version: "encryptionkey@0"
I20260807 03:13:40.965144 5447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:13:40.968713 5463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:40.969986 5447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260807 03:13:40.970157 5447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "6da241f9924c4120a7d0046941405c1d"
format_stamp: "Formatted at 2026-08-07 03:13:40 on dist-test-slave-tm1g"
server_key: "0ce17d418b2891ab857a9c9f799e60a1"
server_key_iv: "1aabbfc1e505b5e2973447ed8e15e3fc"
server_key_version: "encryptionkey@0"
I20260807 03:13:40.970417 5447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:40.984459 5447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:40.988837 5447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:40.989133 5447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:40.990059 5447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:40.991549 5447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:40.991659 5447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:40.991776 5447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:40.991858 5447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:41.025987 5447 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:38111
I20260807 03:13:41.026037 5581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:38111 every 8 connection(s)
I20260807 03:13:41.027325 5447 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:13:41.034247 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 5447
I20260807 03:13:41.034459 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:13:41.035019 17532 external_mini_cluster.cc:1428] Setting key 26cb576ba102bb81af50b6b553b44a8b
Aug 07 03:13:41 dist-test-slave-tm1g krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072420, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.1@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:13:41.049755 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:41.030038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:37795 (local address 127.17.31.62:35001)
0807 03:13:41.030527 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:41.030531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:41.031507 (+ 976us) server_negotiation.cc:407] Connection header received
0807 03:13:41.033367 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:41.033372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:41.033464 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:41.033586 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:41.035710 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:41.036759 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:41.039246 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:41.039650 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:41.039747 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:41.043921 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:41.043926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:41.043929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:41.043960 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:41.046684 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:41.047419 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:41.047425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:41.047427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:41.047557 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:41.048026 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:41.048038 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:41.048040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:41.048296 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:41.048458 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:41.049129 (+ 671us) server_negotiation.cc:299] Negotiation successful
0807 03:13:41.049357 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":259,"threads_started":1}
I20260807 03:13:41.050755 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:41.030643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35001 (local address 127.17.31.1:37795)
0807 03:13:41.031341 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:41.031377 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:41.033092 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:41.033858 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:41.033871 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:41.034676 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:41.035480 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:41.035507 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:41.036955 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:41.036960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:41.039073 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:41.039092 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:41.039882 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:41.039892 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:41.040212 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:41.040940 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:13:41.040975 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:41.043658 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:41.046822 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:41.046831 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:41.046846 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:41.047279 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:41.047683 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:41.047687 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:41.047691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:41.047879 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:41.048428 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:41.048436 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:41.048753 (+ 317us) client_negotiation.cc:769] Sending connection context
0807 03:13:41.049022 (+ 269us) client_negotiation.cc:240] Negotiation successful
0807 03:13:41.049528 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":279,"threads_started":1}
I20260807 03:13:41.052351 5582 heartbeater.cc:347] Connected to a master server at 127.17.31.62:35001
I20260807 03:13:41.052659 5582 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:41.053354 5582 heartbeater.cc:511] Master 127.17.31.62:35001 requested a full tablet report, sending...
I20260807 03:13:41.055480 5383 ts_manager.cc:194] Registered new tserver with Master: 6da241f9924c4120a7d0046941405c1d (127.17.31.1:38111)
I20260807 03:13:41.056885 5383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.1@KRBTEST.COM'} at 127.17.31.1:37795
WARNING: no policy specified for kudu/127.17.31.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
Entry for principal kudu/127.17.31.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab.
I20260807 03:13:41.097123 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35001
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:36033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:41.431453 5588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:41.431706 5588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:41.431851 5588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:41.438920 5588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:41.439018 5588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:41.439172 5588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:13:41.448622 5588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:36033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.2.keytab
--principal=kudu/127.17.31.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:41.449802 5588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:41.451179 5588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:41.464330 5594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:41.464299 5595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:41.464483 5597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:41.465098 5588 server_base.cc:1034] running on GCE node
I20260807 03:13:41.465736 5588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:41.466760 5588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:41.467957 5588 hybrid_clock.cc:648] HybridClock initialized: now 1786072421467905 us; error 61 us; skew 500 ppm
I20260807 03:13:41.468292 5588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:41 dist-test-slave-tm1g krb5kdc[5337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072421, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:41.471447 5588 init.cc:377] Logged in from keytab as kudu/127.17.31.2@KRBTEST.COM (short username kudu)
I20260807 03:13:41.472955 5588 webserver.cc:460] Webserver started at http://127.17.31.2:39503/ using document root <none> and password file <none>
I20260807 03:13:41.473835 5588 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:41.474050 5588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:41.474478 5588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:41.477487 5588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "5e2740669e26426790939c5a32f2dd60"
format_stamp: "Formatted at 2026-08-07 03:13:41 on dist-test-slave-tm1g"
server_key: "ab44098a86e18cf5c7ef1fab68697e66"
server_key_iv: "6a3ce9b2b25772474c64a9cf7a0a17d1"
server_key_version: "encryptionkey@0"
I20260807 03:13:41.478274 5588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "5e2740669e26426790939c5a32f2dd60"
format_stamp: "Formatted at 2026-08-07 03:13:41 on dist-test-slave-tm1g"
server_key: "ab44098a86e18cf5c7ef1fab68697e66"
server_key_iv: "6a3ce9b2b25772474c64a9cf7a0a17d1"
server_key_version: "encryptionkey@0"
I20260807 03:13:41.483433 5588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260807 03:13:41.486706 5604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:41.487926 5588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 03:13:41.488106 5588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "5e2740669e26426790939c5a32f2dd60"
format_stamp: "Formatted at 2026-08-07 03:13:41 on dist-test-slave-tm1g"
server_key: "ab44098a86e18cf5c7ef1fab68697e66"
server_key_iv: "6a3ce9b2b25772474c64a9cf7a0a17d1"
server_key_version: "encryptionkey@0"
I20260807 03:13:41.488302 5588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:41.519724 5588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:41.524175 5588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:41.524483 5588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:41.525460 5588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:41.526829 5588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:41.526934 5588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:41.527097 5588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:41.527170 5588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:41.561528 5588 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:41413
I20260807 03:13:41.561632 5718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:41413 every 8 connection(s)
I20260807 03:13:41.562912 5588 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:13:41.572239 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 5588
I20260807 03:13:41.572446 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:13:41.572909 17532 external_mini_cluster.cc:1428] Setting key 816e23a0accba6dfedc535814243544c
Aug 07 03:13:41 dist-test-slave-tm1g krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072421, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.2@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:13:41.584563 5722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:41.565573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:57699 (local address 127.17.31.62:35001)
0807 03:13:41.566019 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:41.566023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:41.567091 (+ 1068us) server_negotiation.cc:407] Connection header received
0807 03:13:41.569177 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:41.569183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:41.569277 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:41.569394 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:41.570894 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:41.571653 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:41.573347 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:41.574034 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:41.574171 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:41.578290 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:41.578295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:41.578343 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:41.578389 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:41.580975 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:41.581876 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:41.581880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:41.581882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:41.582000 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:41.582494 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:41.582497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:41.582499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:41.582759 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:41.582873 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:41.583858 (+ 985us) server_negotiation.cc:299] Negotiation successful
0807 03:13:41.584233 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":221,"threads_started":1}
I20260807 03:13:41.585610 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:41.566192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35001 (local address 127.17.31.2:57699)
0807 03:13:41.566942 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:41.566980 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:41.568927 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:41.569629 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:41.569637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:41.570117 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:41.570728 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:41.570745 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:41.571809 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:41.571816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:41.573203 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:41.573214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:41.574290 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:41.574341 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:41.574569 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:41.575359 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:13:41.575383 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:41.578072 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:41.581113 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:41.581124 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:41.581140 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:41.581756 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:41.582105 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:41.582112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:41.582119 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:41.582372 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:41.582886 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:41.582896 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:41.583342 (+ 446us) client_negotiation.cc:769] Sending connection context
0807 03:13:41.583636 (+ 294us) client_negotiation.cc:240] Negotiation successful
0807 03:13:41.584181 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":312,"threads_started":1}
I20260807 03:13:41.587045 5719 heartbeater.cc:347] Connected to a master server at 127.17.31.62:35001
I20260807 03:13:41.587337 5719 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:41.588013 5719 heartbeater.cc:511] Master 127.17.31.62:35001 requested a full tablet report, sending...
I20260807 03:13:41.589532 5383 ts_manager.cc:194] Registered new tserver with Master: 5e2740669e26426790939c5a32f2dd60 (127.17.31.2:41413)
I20260807 03:13:41.590873 5383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.2@KRBTEST.COM'} at 127.17.31.2:57699
WARNING: no policy specified for kudu/127.17.31.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
Entry for principal kudu/127.17.31.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab.
I20260807 03:13:41.635070 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35001
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:36033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:41.968914 5727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:41.969213 5727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:41.969388 5727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:41.976504 5727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:41.976640 5727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:41.976824 5727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:13:41.986872 5727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:36033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.3.keytab
--principal=kudu/127.17.31.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:35001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:41.988476 5727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:41.989882 5727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:42.003605 5735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:42.003651 5733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:42.003706 5732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:42.004454 5727 server_base.cc:1034] running on GCE node
I20260807 03:13:42.005008 5727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:42.005920 5727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:42.007143 5727 hybrid_clock.cc:648] HybridClock initialized: now 1786072422007092 us; error 45 us; skew 500 ppm
I20260807 03:13:42.007436 5727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:42 dist-test-slave-tm1g krb5kdc[5337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072422, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:42.010628 5727 init.cc:377] Logged in from keytab as kudu/127.17.31.3@KRBTEST.COM (short username kudu)
I20260807 03:13:42.011960 5727 webserver.cc:460] Webserver started at http://127.17.31.3:36913/ using document root <none> and password file <none>
I20260807 03:13:42.012624 5727 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:42.012769 5727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:42.013087 5727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:42.015890 5727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "7525374892e84ba0b1caff635c19798a"
format_stamp: "Formatted at 2026-08-07 03:13:42 on dist-test-slave-tm1g"
server_key: "72ef1469b457fb3e7545b0d9fdb02582"
server_key_iv: "33f704230baf4ecc8c6edf47d812de88"
server_key_version: "encryptionkey@0"
I20260807 03:13:42.016642 5727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "7525374892e84ba0b1caff635c19798a"
format_stamp: "Formatted at 2026-08-07 03:13:42 on dist-test-slave-tm1g"
server_key: "72ef1469b457fb3e7545b0d9fdb02582"
server_key_iv: "33f704230baf4ecc8c6edf47d812de88"
server_key_version: "encryptionkey@0"
I20260807 03:13:42.021585 5727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260807 03:13:42.024802 5743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:42.026109 5727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260807 03:13:42.026264 5727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "7525374892e84ba0b1caff635c19798a"
format_stamp: "Formatted at 2026-08-07 03:13:42 on dist-test-slave-tm1g"
server_key: "72ef1469b457fb3e7545b0d9fdb02582"
server_key_iv: "33f704230baf4ecc8c6edf47d812de88"
server_key_version: "encryptionkey@0"
I20260807 03:13:42.026486 5727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:42.047729 5727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:42.051755 5727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:42.052038 5727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:42.052987 5727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:42.054266 5727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:42.054361 5727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:42.054517 5727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:42.054570 5727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:42.059841 5582 heartbeater.cc:503] Master 127.17.31.62:35001 was elected leader, sending a full tablet report...
I20260807 03:13:42.087885 5727 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:36905
I20260807 03:13:42.087986 5856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:36905 every 8 connection(s)
I20260807 03:13:42.089159 5727 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:13:42.091058 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 5727
I20260807 03:13:42.091248 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:13:42.091780 17532 external_mini_cluster.cc:1428] Setting key 58c53e439e7dd1145f6f9af3d79a0fa8
Aug 07 03:13:42 dist-test-slave-tm1g krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072422, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.3@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:13:42.110046 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.091978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:57803 (local address 127.17.31.62:35001)
0807 03:13:42.092446 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:42.092452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:42.093708 (+ 1256us) server_negotiation.cc:407] Connection header received
0807 03:13:42.095844 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:42.095849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:42.095932 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:42.096062 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:42.097528 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.098420 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.099972 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.100384 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.100520 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.104538 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:42.104542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:42.104546 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:42.104576 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:42.107452 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:42.108135 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:42.108139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:42.108141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:42.108269 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:42.108625 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:42.108628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:42.108629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:42.108900 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:42.109025 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:42.109608 (+ 583us) server_negotiation.cc:299] Negotiation successful
0807 03:13:42.109761 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":217,"threads_started":1}
I20260807 03:13:42.111057 5860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.092625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:35001 (local address 127.17.31.3:57803)
0807 03:13:42.093533 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:42.093595 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:42.095572 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:42.096299 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:42.096308 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:42.096833 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:42.097365 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:42.097381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.098570 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.098575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:42.099808 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:42.099819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.100603 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.100609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:42.100858 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.101546 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:13:42.101567 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:42.104313 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:42.107573 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:42.107579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:42.107589 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:42.108015 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:42.108366 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:13:42.108369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:13:42.108372 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:13:42.108533 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:13:42.109005 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:42.109010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:42.109283 (+ 273us) client_negotiation.cc:769] Sending connection context
0807 03:13:42.109506 (+ 223us) client_negotiation.cc:240] Negotiation successful
0807 03:13:42.109850 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":387,"threads_started":1}
I20260807 03:13:42.112418 5857 heartbeater.cc:347] Connected to a master server at 127.17.31.62:35001
I20260807 03:13:42.112730 5857 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:42.113404 5857 heartbeater.cc:511] Master 127.17.31.62:35001 requested a full tablet report, sending...
I20260807 03:13:42.114765 5383 ts_manager.cc:194] Registered new tserver with Master: 7525374892e84ba0b1caff635c19798a (127.17.31.3:36905)
I20260807 03:13:42.115561 5383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.31.3@KRBTEST.COM'} at 127.17.31.3:57803
I20260807 03:13:42.119077 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:13:42.135301 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.123042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.17.31.62:35001)
0807 03:13:42.123249 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:42.123257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:42.123731 (+ 474us) server_negotiation.cc:407] Connection header received
0807 03:13:42.123848 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:42.123851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:42.123921 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:42.123999 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:42.125343 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.126054 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.127423 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.127837 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.127935 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.130081 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:42.130085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:42.130088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:42.130135 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:42.132911 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:42.133493 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:42.133497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:42.133499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:42.133624 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:42.133973 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:42.133978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:42.133980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:42.134274 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:42.134546 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:42.134801 (+ 255us) server_negotiation.cc:299] Negotiation successful
0807 03:13:42.134955 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 03:13:42.138779 5383 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:13:42.145663 5383 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:13:42.184702 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.157084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:38111 (local address 127.0.0.1:43050)
0807 03:13:42.168866 (+ 11782us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:42.168923 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:42.169124 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:42.176872 (+ 7748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:42.176881 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:42.176913 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:42.177413 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:42.177425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.179236 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.179241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:42.181695 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:42.181709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.183620 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.183627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:42.183839 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.183909 (+ 70us) client_negotiation.cc:769] Sending connection context
0807 03:13:42.184038 (+ 129us) client_negotiation.cc:240] Negotiation successful
0807 03:13:42.184238 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11469,"mutex_wait_us":90,"thread_start_us":12223,"threads_started":1}
I20260807 03:13:42.184721 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.157461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43050 (local address 127.17.31.1:38111)
0807 03:13:42.176228 (+ 18767us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:42.176237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:42.176267 (+ 30us) server_negotiation.cc:407] Connection header received
0807 03:13:42.176393 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:42.176402 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:42.176629 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:42.177657 (+ 1028us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:42.177886 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.179104 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.181856 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.183401 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.183524 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.183746 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:42.184241 (+ 495us) server_negotiation.cc:299] Negotiation successful
0807 03:13:42.184422 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18587,"thread_start_us":18417,"threads_started":1}
I20260807 03:13:42.185544 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.157253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:36905 (local address 127.0.0.1:59646)
0807 03:13:42.174737 (+ 17484us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:42.174758 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:42.174912 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:42.177414 (+ 2502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:42.177419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:42.177450 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:42.177961 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:42.177977 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.180072 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.180077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:42.182435 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:42.182450 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.184466 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.184472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:42.184647 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.184679 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 03:13:42.184773 (+ 94us) client_negotiation.cc:240] Negotiation successful
0807 03:13:42.185224 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17382,"thread_start_us":18255,"threads_started":1}
I20260807 03:13:42.185528 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.159875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.17.31.2:41413)
0807 03:13:42.175321 (+ 15446us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:42.175334 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:42.175923 (+ 589us) server_negotiation.cc:407] Connection header received
0807 03:13:42.176330 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:42.176343 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:42.176607 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:42.176822 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:42.178008 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.179655 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.182257 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.183813 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.183942 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.184125 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:42.185011 (+ 886us) server_negotiation.cc:299] Negotiation successful
0807 03:13:42.185177 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15241,"thread_start_us":14656,"threads_started":1}
I20260807 03:13:42.185873 5872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.159184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:41413 (local address 127.0.0.1:54920)
0807 03:13:42.175784 (+ 16600us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:42.175823 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:42.176044 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:42.177078 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:42.177083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:42.177113 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:42.177838 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:42.177850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.179794 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.179801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:42.182054 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:42.182067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.184029 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.184039 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:42.184197 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.184227 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 03:13:42.184330 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 03:13:42.184476 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16477,"mutex_wait_us":39,"thread_start_us":15181,"threads_started":1}
I20260807 03:13:42.186743 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.157484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.17.31.3:36905)
0807 03:13:42.176873 (+ 19389us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:42.176881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:42.176907 (+ 26us) server_negotiation.cc:407] Connection header received
0807 03:13:42.176998 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:42.177004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:42.177204 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:42.178072 (+ 868us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:42.178410 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.179926 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.182616 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.183997 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.184117 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.184325 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:42.186172 (+ 1847us) server_negotiation.cc:299] Negotiation successful
0807 03:13:42.186375 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19008,"thread_start_us":18874,"threads_started":1}
I20260807 03:13:42.189047 5512 tablet_service.cc:1467] Processing CreateTablet for tablet c790fbdb149740739bc625239a136d59 (DEFAULT_TABLE table=test-table [id=690b63525f9e484888c293beaea41efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:42.189499 5792 tablet_service.cc:1467] Processing CreateTablet for tablet c790fbdb149740739bc625239a136d59 (DEFAULT_TABLE table=test-table [id=690b63525f9e484888c293beaea41efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:42.189816 5654 tablet_service.cc:1467] Processing CreateTablet for tablet c790fbdb149740739bc625239a136d59 (DEFAULT_TABLE table=test-table [id=690b63525f9e484888c293beaea41efb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:42.190680 5512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c790fbdb149740739bc625239a136d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:42.191151 5654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c790fbdb149740739bc625239a136d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:42.191349 5792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c790fbdb149740739bc625239a136d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:42.203083 5878 tablet_bootstrap.cc:492] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a: Bootstrap starting.
I20260807 03:13:42.205339 5879 tablet_bootstrap.cc:492] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d: Bootstrap starting.
I20260807 03:13:42.206746 5878 tablet_bootstrap.cc:654] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:42.206897 5880 tablet_bootstrap.cc:492] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60: Bootstrap starting.
I20260807 03:13:42.208168 5878 log.cc:824] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:42.208885 5879 tablet_bootstrap.cc:654] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:42.209959 5879 log.cc:824] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:42.210186 5880 tablet_bootstrap.cc:654] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:42.211459 5880 log.cc:824] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:42.211686 5878 tablet_bootstrap.cc:492] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a: No bootstrap required, opened a new log
I20260807 03:13:42.211961 5878 ts_tablet_manager.cc:1397] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 03:13:42.212971 5879 tablet_bootstrap.cc:492] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d: No bootstrap required, opened a new log
I20260807 03:13:42.213209 5879 ts_tablet_manager.cc:1397] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260807 03:13:42.213980 5880 tablet_bootstrap.cc:492] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60: No bootstrap required, opened a new log
I20260807 03:13:42.214217 5880 ts_tablet_manager.cc:1397] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260807 03:13:42.221535 5880 raft_consensus.cc:348] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da241f9924c4120a7d0046941405c1d" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38111 } } peers { permanent_uuid: "7525374892e84ba0b1caff635c19798a" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36905 } } peers { permanent_uuid: "5e2740669e26426790939c5a32f2dd60" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41413 } }
I20260807 03:13:42.221948 5880 raft_consensus.cc:374] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:42.222050 5880 raft_consensus.cc:729] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e2740669e26426790939c5a32f2dd60, State: Initialized, Role: FOLLOWER
I20260807 03:13:42.222280 5878 raft_consensus.cc:348] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da241f9924c4120a7d0046941405c1d" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38111 } } peers { permanent_uuid: "7525374892e84ba0b1caff635c19798a" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36905 } } peers { permanent_uuid: "5e2740669e26426790939c5a32f2dd60" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41413 } }
I20260807 03:13:42.222682 5878 raft_consensus.cc:374] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:42.222772 5878 raft_consensus.cc:729] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7525374892e84ba0b1caff635c19798a, State: Initialized, Role: FOLLOWER
I20260807 03:13:42.222683 5880 consensus_queue.cc:260] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da241f9924c4120a7d0046941405c1d" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38111 } } peers { permanent_uuid: "7525374892e84ba0b1caff635c19798a" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36905 } } peers { permanent_uuid: "5e2740669e26426790939c5a32f2dd60" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41413 } }
I20260807 03:13:42.223364 5878 consensus_queue.cc:260] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da241f9924c4120a7d0046941405c1d" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38111 } } peers { permanent_uuid: "7525374892e84ba0b1caff635c19798a" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36905 } } peers { permanent_uuid: "5e2740669e26426790939c5a32f2dd60" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41413 } }
I20260807 03:13:42.223610 5879 raft_consensus.cc:348] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da241f9924c4120a7d0046941405c1d" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38111 } } peers { permanent_uuid: "7525374892e84ba0b1caff635c19798a" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36905 } } peers { permanent_uuid: "5e2740669e26426790939c5a32f2dd60" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41413 } }
I20260807 03:13:42.223892 5719 heartbeater.cc:503] Master 127.17.31.62:35001 was elected leader, sending a full tablet report...
I20260807 03:13:42.224050 5879 raft_consensus.cc:374] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:42.224166 5879 raft_consensus.cc:729] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6da241f9924c4120a7d0046941405c1d, State: Initialized, Role: FOLLOWER
I20260807 03:13:42.224606 5880 ts_tablet_manager.cc:1428] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260807 03:13:42.224833 5879 consensus_queue.cc:260] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da241f9924c4120a7d0046941405c1d" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38111 } } peers { permanent_uuid: "7525374892e84ba0b1caff635c19798a" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36905 } } peers { permanent_uuid: "5e2740669e26426790939c5a32f2dd60" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41413 } }
I20260807 03:13:42.225937 5857 heartbeater.cc:503] Master 127.17.31.62:35001 was elected leader, sending a full tablet report...
I20260807 03:13:42.226778 5878 ts_tablet_manager.cc:1428] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260807 03:13:42.228281 5879 ts_tablet_manager.cc:1428] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260807 03:13:42.279737 5583 tablet.cc:2369] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:13:42.286545 5884 raft_consensus.cc:482] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:13:42.286796 5884 raft_consensus.cc:504] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da241f9924c4120a7d0046941405c1d" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38111 } } peers { permanent_uuid: "7525374892e84ba0b1caff635c19798a" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36905 } } peers { permanent_uuid: "5e2740669e26426790939c5a32f2dd60" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41413 } }
I20260807 03:13:42.288419 5884 leader_election.cc:288] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6da241f9924c4120a7d0046941405c1d (127.17.31.1:38111), 7525374892e84ba0b1caff635c19798a (127.17.31.3:36905)
I20260807 03:13:42.298030 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.288776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:47055 (local address 127.17.31.1:38111)
0807 03:13:42.289034 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:42.289043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:42.289914 (+ 871us) server_negotiation.cc:407] Connection header received
0807 03:13:42.290058 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:42.290063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:42.290133 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:42.290239 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:42.291426 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.292514 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.295406 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.296645 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.296775 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.296880 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:42.297635 (+ 755us) server_negotiation.cc:299] Negotiation successful
0807 03:13:42.297767 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 03:13:42.298030 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.288996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:53319 (local address 127.17.31.3:36905)
0807 03:13:42.289256 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:42.289265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:42.289740 (+ 475us) server_negotiation.cc:407] Connection header received
0807 03:13:42.290041 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:42.290046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:42.290123 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:42.290218 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:42.291134 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.292362 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.295405 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.296642 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.296775 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.296879 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:42.297635 (+ 756us) server_negotiation.cc:299] Negotiation successful
0807 03:13:42.297767 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 03:13:42.298367 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.288654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:36905 (local address 127.17.31.2:53319)
0807 03:13:42.289626 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:42.289646 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:42.289819 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:42.290324 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:42.290329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:42.290354 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:42.290956 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:42.290969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.292497 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.292501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:42.295242 (+ 2741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:42.295252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.297006 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.297010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:42.297351 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.297483 (+ 132us) client_negotiation.cc:769] Sending connection context
0807 03:13:42.297589 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 03:13:42.297805 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":293,"thread_start_us":576,"threads_started":1}
I20260807 03:13:42.298405 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.288657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:38111 (local address 127.17.31.2:47055)
0807 03:13:42.289809 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:42.289823 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:42.289927 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:42.290450 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:42.290454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:42.290493 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:42.291183 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:42.291199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.292686 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.292693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:42.295238 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:42.295252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.297003 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.297010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:42.297352 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.297488 (+ 136us) client_negotiation.cc:769] Sending connection context
0807 03:13:42.297589 (+ 101us) client_negotiation.cc:240] Negotiation successful
0807 03:13:42.297804 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"mutex_wait_us":454,"thread_start_us":717,"threads_started":1}
I20260807 03:13:42.299459 5533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c790fbdb149740739bc625239a136d59" candidate_uuid: "5e2740669e26426790939c5a32f2dd60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6da241f9924c4120a7d0046941405c1d" is_pre_election: true
I20260807 03:13:42.299458 5812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c790fbdb149740739bc625239a136d59" candidate_uuid: "5e2740669e26426790939c5a32f2dd60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7525374892e84ba0b1caff635c19798a" is_pre_election: true
I20260807 03:13:42.299928 5533 raft_consensus.cc:2454] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e2740669e26426790939c5a32f2dd60 in term 0.
I20260807 03:13:42.299952 5812 raft_consensus.cc:2454] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e2740669e26426790939c5a32f2dd60 in term 0.
I20260807 03:13:42.300652 5608 leader_election.cc:302] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5e2740669e26426790939c5a32f2dd60, 6da241f9924c4120a7d0046941405c1d, 7525374892e84ba0b1caff635c19798a; no voters:
I20260807 03:13:42.301051 5884 raft_consensus.cc:2781] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:13:42.301205 5884 raft_consensus.cc:482] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:13:42.301316 5884 raft_consensus.cc:3037] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:42.303575 5884 raft_consensus.cc:504] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da241f9924c4120a7d0046941405c1d" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38111 } } peers { permanent_uuid: "7525374892e84ba0b1caff635c19798a" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36905 } } peers { permanent_uuid: "5e2740669e26426790939c5a32f2dd60" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41413 } }
I20260807 03:13:42.304220 5884 leader_election.cc:288] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [CANDIDATE]: Term 1 election: Requested vote from peers 6da241f9924c4120a7d0046941405c1d (127.17.31.1:38111), 7525374892e84ba0b1caff635c19798a (127.17.31.3:36905)
I20260807 03:13:42.304536 5533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c790fbdb149740739bc625239a136d59" candidate_uuid: "5e2740669e26426790939c5a32f2dd60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6da241f9924c4120a7d0046941405c1d"
I20260807 03:13:42.304725 5533 raft_consensus.cc:3037] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:42.304714 5812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c790fbdb149740739bc625239a136d59" candidate_uuid: "5e2740669e26426790939c5a32f2dd60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7525374892e84ba0b1caff635c19798a"
I20260807 03:13:42.304896 5812 raft_consensus.cc:3037] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:42.306972 5533 raft_consensus.cc:2454] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e2740669e26426790939c5a32f2dd60 in term 1.
I20260807 03:13:42.307205 5812 raft_consensus.cc:2454] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e2740669e26426790939c5a32f2dd60 in term 1.
I20260807 03:13:42.307394 5605 leader_election.cc:302] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e2740669e26426790939c5a32f2dd60, 6da241f9924c4120a7d0046941405c1d; no voters:
I20260807 03:13:42.307734 5884 raft_consensus.cc:2781] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:42.308161 5884 raft_consensus.cc:686] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [term 1 LEADER]: Becoming Leader. State: Replica: 5e2740669e26426790939c5a32f2dd60, State: Running, Role: LEADER
I20260807 03:13:42.308637 5884 consensus_queue.cc:237] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da241f9924c4120a7d0046941405c1d" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38111 } } peers { permanent_uuid: "7525374892e84ba0b1caff635c19798a" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36905 } } peers { permanent_uuid: "5e2740669e26426790939c5a32f2dd60" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41413 } }
I20260807 03:13:42.313216 5383 catalog_manager.cc:5495] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e2740669e26426790939c5a32f2dd60 (127.17.31.2). New cstate: current_term: 1 leader_uuid: "5e2740669e26426790939c5a32f2dd60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da241f9924c4120a7d0046941405c1d" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7525374892e84ba0b1caff635c19798a" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e2740669e26426790939c5a32f2dd60" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 41413 } health_report { overall_health: HEALTHY } } }
W20260807 03:13:42.314925 5720 tablet.cc:2369] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:13:42.331965 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.326900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54930 (local address 127.17.31.2:41413)
0807 03:13:42.327114 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:42.327118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:42.327135 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:13:42.327282 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:42.327286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:42.327354 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:42.327468 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:13:42.327788 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.328515 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.329822 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.330275 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.330420 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.330931 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:13:42.331409 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:13:42.331525 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:42.331606 (+ 81us) server_negotiation.cc:299] Negotiation successful
0807 03:13:42.331666 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 03:13:42.339684 5812 raft_consensus.cc:1261] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a [term 1 FOLLOWER]: Refusing update from remote peer 5e2740669e26426790939c5a32f2dd60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:13:42.339715 5533 raft_consensus.cc:1261] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d [term 1 FOLLOWER]: Refusing update from remote peer 5e2740669e26426790939c5a32f2dd60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:13:42.340656 5890 consensus_queue.cc:1035] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6da241f9924c4120a7d0046941405c1d" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 38111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 03:13:42.340847 5858 tablet.cc:2369] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:13:42.341097 5884 consensus_queue.cc:1035] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7525374892e84ba0b1caff635c19798a" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 36905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:13:42.355118 5893 mvcc.cc:205] Tried to move back new op lower bound from 7315752641894244352 to 7315752641779699712. Current Snapshot: MvccSnapshot[applied={T|T < 7315752641894244352 or (T in {7315752641894244352})}]
I20260807 03:13:42.363668 5381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:13:42.364013 5381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:13:42.368450 5381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8bed7e39392c41c8b6a86b5b35eafc9e: Sending DeleteTablet for 3 replicas of tablet c790fbdb149740739bc625239a136d59
I20260807 03:13:42.369765 5512 tablet_service.cc:1514] Processing DeleteTablet for tablet c790fbdb149740739bc625239a136d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:42 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:43050
I20260807 03:13:42.370249 5792 tablet_service.cc:1514] Processing DeleteTablet for tablet c790fbdb149740739bc625239a136d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:42 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:59646
I20260807 03:13:42.370364 5654 tablet_service.cc:1514] Processing DeleteTablet for tablet c790fbdb149740739bc625239a136d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:42 UTC) from {username='kudu', principal='kudu/127.17.31.62@KRBTEST.COM'} at 127.0.0.1:54920
I20260807 03:13:42.371619 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 5447
I20260807 03:13:42.382961 5905 tablet_replica.cc:330] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60: stopping tablet replica
I20260807 03:13:42.383534 5905 raft_consensus.cc:2229] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:13:42.384305 5905 raft_consensus.cc:2258] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:42.385154 5903 tablet_replica.cc:330] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d: stopping tablet replica
I20260807 03:13:42.385671 5903 raft_consensus.cc:2229] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:13:42.386252 5903 raft_consensus.cc:2258] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:42.388764 5903 ts_tablet_manager.cc:1905] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:42.390587 5904 tablet_replica.cc:330] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a: stopping tablet replica
I20260807 03:13:42.390949 5904 raft_consensus.cc:2229] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:13:42.391746 5905 ts_tablet_manager.cc:1905] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:42.395164 5904 raft_consensus.cc:2258] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:42.396327 5903 ts_tablet_manager.cc:1918] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:42.396498 5903 log.cc:1196] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/c790fbdb149740739bc625239a136d59
I20260807 03:13:42.397130 5903 ts_tablet_manager.cc:1939] T c790fbdb149740739bc625239a136d59 P 6da241f9924c4120a7d0046941405c1d: Deleting consensus metadata
I20260807 03:13:42.397993 5904 ts_tablet_manager.cc:1905] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:42.398967 5367 catalog_manager.cc:4841] TS 6da241f9924c4120a7d0046941405c1d (127.17.31.1:38111): tablet c790fbdb149740739bc625239a136d59 (table test-table [id=690b63525f9e484888c293beaea41efb]) successfully deleted
I20260807 03:13:42.400441 5905 ts_tablet_manager.cc:1918] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:42.400614 5905 log.cc:1196] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/c790fbdb149740739bc625239a136d59
I20260807 03:13:42.401150 5905 ts_tablet_manager.cc:1939] T c790fbdb149740739bc625239a136d59 P 5e2740669e26426790939c5a32f2dd60: Deleting consensus metadata
I20260807 03:13:42.402846 5369 catalog_manager.cc:4841] TS 5e2740669e26426790939c5a32f2dd60 (127.17.31.2:41413): tablet c790fbdb149740739bc625239a136d59 (table test-table [id=690b63525f9e484888c293beaea41efb]) successfully deleted
I20260807 03:13:42.405125 5904 ts_tablet_manager.cc:1918] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:42.405311 5904 log.cc:1196] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/c790fbdb149740739bc625239a136d59
Aug 07 03:13:42 dist-test-slave-tm1g krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.1@KRBTEST.COM
I20260807 03:13:42.405755 5904 ts_tablet_manager.cc:1939] T c790fbdb149740739bc625239a136d59 P 7525374892e84ba0b1caff635c19798a: Deleting consensus metadata
I20260807 03:13:42.407187 5370 catalog_manager.cc:4841] TS 7525374892e84ba0b1caff635c19798a (127.17.31.3:36905): tablet c790fbdb149740739bc625239a136d59 (table test-table [id=690b63525f9e484888c293beaea41efb]) successfully deleted
I20260807 03:13:42.412945 5873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.373988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.17.31.1:38111)
0807 03:13:42.374222 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:42.374231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:42.387912 (+ 13681us) server_negotiation.cc:407] Connection header received
0807 03:13:42.388373 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:42.388378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:42.388462 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:42.388553 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:42.391720 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.392864 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.395768 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.396415 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.396544 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.407042 (+ 10498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:42.407060 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:42.407078 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:42.407122 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:42.410080 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:42.410937 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:42.410944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:42.410946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:42.411080 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:42.411513 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:42.411516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:42.411518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:42.411851 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:42.412079 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:42.412462 (+ 383us) server_negotiation.cc:299] Negotiation successful
0807 03:13:42.412624 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 03:13:42.413738 5576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:42.520535 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 5447
I20260807 03:13:42.546720 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 5588
Aug 07 03:13:42 dist-test-slave-tm1g krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.2@KRBTEST.COM
I20260807 03:13:42.564517 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.547603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54934 (local address 127.17.31.2:41413)
0807 03:13:42.547783 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:42.547788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:42.547805 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:13:42.547998 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:42.548002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:42.548066 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:42.548145 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:42.549800 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.550729 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.552661 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.553127 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.553277 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.557809 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:42.557822 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:42.557838 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:42.557882 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:42.560988 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:42.561861 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:42.561870 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:42.561874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:42.562069 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:42.562669 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:42.562674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:42.562678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:42.563172 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:42.563564 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:42.563860 (+ 296us) server_negotiation.cc:299] Negotiation successful
0807 03:13:42.564108 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 03:13:42.565301 5714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:42.682958 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 5588
I20260807 03:13:42.705303 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 5727
Aug 07 03:13:42 dist-test-slave-tm1g krb5kdc[5337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.3@KRBTEST.COM
I20260807 03:13:42.720585 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:42.706085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.17.31.3:36905)
0807 03:13:42.706277 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:42.706285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:42.706343 (+ 58us) server_negotiation.cc:407] Connection header received
0807 03:13:42.706579 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:42.706584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:42.706660 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:42.706767 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:42.708131 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.708875 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.710195 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:42.710878 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:42.711023 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:42.715505 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:42.715518 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:42.715533 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:42.715564 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:42.718052 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:42.718884 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:42.718891 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:42.718894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:42.719029 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:42.719369 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:42.719372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:42.719373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:42.719707 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:42.719917 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:42.720174 (+ 257us) server_negotiation.cc:299] Negotiation successful
0807 03:13:42.720334 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 03:13:42.721199 5852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:42.810432 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 5727
I20260807 03:13:42.828895 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 5351
I20260807 03:13:42.830057 5413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:42.940479 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 5351
2026-08-07T03:13:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4338 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-07T03:13:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:13:43Z Disabled control of system clock
I20260807 03:13:43.024021 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:37215
--webserver_interface=127.17.31.62
--webserver_port=0
--builtin_ntp_servers=127.17.31.20:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:37215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 03:13:43.347865 5915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:43.348129 5915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:43.348296 5915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:43.355329 5915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:13:43.355440 5915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:43.355505 5915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:43.355610 5915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:13:43.355690 5915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:13:43.364395 5915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:37215
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:37215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:43.365624 5915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:43.367326 5915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:43.378669 5920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:43.378724 5923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:43.378719 5921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:43.380110 5915 server_base.cc:1034] running on GCE node
I20260807 03:13:43.380824 5915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:43.382089 5915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:43.383308 5915 hybrid_clock.cc:648] HybridClock initialized: now 1786072423383236 us; error 85 us; skew 500 ppm
I20260807 03:13:43.383656 5915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:13:43.385794 5915 webserver.cc:460] Webserver started at http://127.17.31.62:44443/ using document root <none> and password file <none>
I20260807 03:13:43.386525 5915 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:43.386655 5915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:43.386973 5915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:43.389741 5915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "7305b13923a24d1eb90932030bedd699"
format_stamp: "Formatted at 2026-08-07 03:13:43 on dist-test-slave-tm1g"
server_key: "f5a925ef403ed1bae504d90759f0a8dc"
server_key_iv: "1676d1d2bf4e7d2788d8008dfc30d26d"
server_key_version: "encryptionkey@0"
I20260807 03:13:43.390595 5915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "7305b13923a24d1eb90932030bedd699"
format_stamp: "Formatted at 2026-08-07 03:13:43 on dist-test-slave-tm1g"
server_key: "f5a925ef403ed1bae504d90759f0a8dc"
server_key_iv: "1676d1d2bf4e7d2788d8008dfc30d26d"
server_key_version: "encryptionkey@0"
I20260807 03:13:43.395509 5915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 03:13:43.398763 5930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:43.400035 5915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 03:13:43.400213 5915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "7305b13923a24d1eb90932030bedd699"
format_stamp: "Formatted at 2026-08-07 03:13:43 on dist-test-slave-tm1g"
server_key: "f5a925ef403ed1bae504d90759f0a8dc"
server_key_iv: "1676d1d2bf4e7d2788d8008dfc30d26d"
server_key_version: "encryptionkey@0"
I20260807 03:13:43.400434 5915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:43.415633 5915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:43.416357 5915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:43.416636 5915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:43.434695 5915 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:37215
I20260807 03:13:43.434726 5982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:37215 every 8 connection(s)
I20260807 03:13:43.435964 5915 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:13:43.439512 5983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:43.446054 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 5915
I20260807 03:13:43.446230 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:13:43.446691 17532 external_mini_cluster.cc:1428] Setting key df830fc56a14fb90cf2ef32d73da82f6
I20260807 03:13:43.449985 5983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699: Bootstrap starting.
I20260807 03:13:43.453289 5983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:43.454416 5983 log.cc:824] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:43.457444 5983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699: No bootstrap required, opened a new log
I20260807 03:13:43.460053 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:43.448529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.17.31.62:37215)
0807 03:13:43.449465 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:43.449475 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:43.449513 (+ 38us) server_negotiation.cc:407] Connection header received
0807 03:13:43.451312 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:43.451336 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:43.451778 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:43.452078 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:43.452760 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:43.454100 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:43.455800 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:43.456444 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:43.456571 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:43.457780 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:43.457787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:43.457799 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:43.458355 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:43.458452 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:43.458471 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:43.458554 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:43.458803 (+ 249us) server_negotiation.cc:299] Negotiation successful
0807 03:13:43.459122 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":326,"threads_started":1}
I20260807 03:13:43.465583 5983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7305b13923a24d1eb90932030bedd699" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37215 } }
I20260807 03:13:43.465925 5983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:43.466059 5983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7305b13923a24d1eb90932030bedd699, State: Initialized, Role: FOLLOWER
I20260807 03:13:43.466848 5983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7305b13923a24d1eb90932030bedd699" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37215 } }
I20260807 03:13:43.467113 5988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:13:43.467370 5988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7305b13923a24d1eb90932030bedd699" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37215 } }
I20260807 03:13:43.468015 5988 leader_election.cc:302] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7305b13923a24d1eb90932030bedd699; no voters:
I20260807 03:13:43.468770 5988 leader_election.cc:288] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260807 03:13:43.468902 5989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:13:43.469099 5989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:13:43.469257 5989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:43.471446 5989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7305b13923a24d1eb90932030bedd699" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37215 } }
I20260807 03:13:43.471715 5989 leader_election.cc:302] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7305b13923a24d1eb90932030bedd699; no voters:
I20260807 03:13:43.468878 5983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:13:43.471910 5989 leader_election.cc:288] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:13:43.471998 5988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:43.472146 5988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [term 1 LEADER]: Becoming Leader. State: Replica: 7305b13923a24d1eb90932030bedd699, State: Running, Role: LEADER
I20260807 03:13:43.472582 5988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7305b13923a24d1eb90932030bedd699" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37215 } }
I20260807 03:13:43.473857 5983 raft_consensus.cc:465] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [term 1 LEADER]: Not starting leader election -- already a leader
I20260807 03:13:43.474274 5988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7305b13923a24d1eb90932030bedd699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7305b13923a24d1eb90932030bedd699" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37215 } } }
I20260807 03:13:43.474525 5989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7305b13923a24d1eb90932030bedd699. Latest consensus state: current_term: 1 leader_uuid: "7305b13923a24d1eb90932030bedd699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7305b13923a24d1eb90932030bedd699" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 37215 } } }
I20260807 03:13:43.474763 5988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:43.474761 5989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:43.475782 5983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:13:43.476629 5991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:13:43.481791 5991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:13:43.490988 5991 catalog_manager.cc:1347] Generated new cluster ID: da8a4f7a063e438bb69a1daaa648b2c5
I20260807 03:13:43.491088 5991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:13:43.508935 5991 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:13:43.510645 5991 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:13:43.524034 5991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699: Generated new TSK 0
I20260807 03:13:43.525003 5991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:13:43.534927 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.1:0
--local_ip_for_outbound_sockets=127.17.31.1
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37215
--builtin_ntp_servers=127.17.31.20:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 03:13:43.876739 6006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:43.877012 6006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:43.877172 6006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:43.884438 6006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:43.884548 6006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:43.884727 6006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.1
I20260807 03:13:43.893488 6006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.31.1
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.1
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:43.894759 6006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:43.896498 6006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:43.909567 6015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:43.909693 6011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:43.909518 6013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:43.910599 6006 server_base.cc:1034] running on GCE node
I20260807 03:13:43.911170 6006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:43.912055 6006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:43.913276 6006 hybrid_clock.cc:648] HybridClock initialized: now 1786072423913235 us; error 56 us; skew 500 ppm
I20260807 03:13:43.913600 6006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:13:43.915902 6006 webserver.cc:460] Webserver started at http://127.17.31.1:32829/ using document root <none> and password file <none>
I20260807 03:13:43.916589 6006 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:43.916741 6006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:43.917081 6006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:43.919884 6006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/data/instance:
uuid: "4ad0975eff90445493314f9a660d1388"
format_stamp: "Formatted at 2026-08-07 03:13:43 on dist-test-slave-tm1g"
server_key: "f753f3847ef69f7690ed7f9949fa2d32"
server_key_iv: "92c1bd5b98d9a9ce4f245e154c17e925"
server_key_version: "encryptionkey@0"
I20260807 03:13:43.920792 6006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance:
uuid: "4ad0975eff90445493314f9a660d1388"
format_stamp: "Formatted at 2026-08-07 03:13:43 on dist-test-slave-tm1g"
server_key: "f753f3847ef69f7690ed7f9949fa2d32"
server_key_iv: "92c1bd5b98d9a9ce4f245e154c17e925"
server_key_version: "encryptionkey@0"
I20260807 03:13:43.925626 6006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260807 03:13:43.928970 6021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:43.930181 6006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 03:13:43.930384 6006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/wal
uuid: "4ad0975eff90445493314f9a660d1388"
format_stamp: "Formatted at 2026-08-07 03:13:43 on dist-test-slave-tm1g"
server_key: "f753f3847ef69f7690ed7f9949fa2d32"
server_key_iv: "92c1bd5b98d9a9ce4f245e154c17e925"
server_key_version: "encryptionkey@0"
I20260807 03:13:43.930598 6006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:43.949527 6006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:43.950284 6006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:43.950618 6006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:43.951558 6006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:43.952976 6006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:43.953106 6006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:43.953229 6006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:43.953320 6006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:43.987656 6006 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.1:44799
I20260807 03:13:43.987744 6138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.1:44799 every 8 connection(s)
I20260807 03:13:43.988992 6006 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/data/info.pb
I20260807 03:13:43.995545 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 6006
I20260807 03:13:43.995751 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/wal/instance
I20260807 03:13:43.996191 17532 external_mini_cluster.cc:1428] Setting key dd79d9ae54dcb55cbac755b363d00718
I20260807 03:13:43.999395 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.2:0
--local_ip_for_outbound_sockets=127.17.31.2
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37215
--builtin_ntp_servers=127.17.31.20:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 03:13:44.003857 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:43.991920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:42045 (local address 127.17.31.62:37215)
0807 03:13:43.992374 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:43.992380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:43.994088 (+ 1708us) server_negotiation.cc:407] Connection header received
0807 03:13:43.996450 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:43.996455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:43.996541 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:43.996660 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:43.997870 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:43.999059 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.000676 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.001412 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.001553 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:44.002925 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:44.002935 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:44.002942 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:44.002967 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:44.003033 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:44.003070 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:44.003164 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:44.003414 (+ 250us) server_negotiation.cc:299] Negotiation successful
0807 03:13:44.003514 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":246,"threads_started":1}
I20260807 03:13:44.004889 6143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:43.992643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:37215 (local address 127.17.31.1:42045)
0807 03:13:43.993899 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:43.993966 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:43.996155 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:43.996887 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:43.996895 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:43.996976 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:43.997716 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:43.997740 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:43.999206 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:43.999210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:44.000515 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:44.000527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.001626 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.001630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:44.001851 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:44.002593 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:13:44.002595 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:44.002633 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:13:44.002711 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:44.003165 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:44.003173 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:44.003178 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 03:13:44.003381 (+ 203us) client_negotiation.cc:240] Negotiation successful
0807 03:13:44.003769 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":422,"threads_started":1}
I20260807 03:13:44.006457 6139 heartbeater.cc:347] Connected to a master server at 127.17.31.62:37215
I20260807 03:13:44.006755 6139 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:44.007491 6139 heartbeater.cc:511] Master 127.17.31.62:37215 requested a full tablet report, sending...
I20260807 03:13:44.009493 5947 ts_manager.cc:194] Registered new tserver with Master: 4ad0975eff90445493314f9a660d1388 (127.17.31.1:44799)
I20260807 03:13:44.011075 5947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.31.1:42045
W20260807 03:13:44.338660 6144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:44.338960 6144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:44.339098 6144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:44.346227 6144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:44.346346 6144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:44.346524 6144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.2
I20260807 03:13:44.355439 6144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.31.2
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.2
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:44.356796 6144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:44.358669 6144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:44.371511 6150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:44.371465 6152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:44.371551 6149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:44.372627 6144 server_base.cc:1034] running on GCE node
I20260807 03:13:44.373199 6144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:44.374106 6144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:44.375339 6144 hybrid_clock.cc:648] HybridClock initialized: now 1786072424375303 us; error 55 us; skew 500 ppm
I20260807 03:13:44.375659 6144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:13:44.378021 6144 webserver.cc:460] Webserver started at http://127.17.31.2:42829/ using document root <none> and password file <none>
I20260807 03:13:44.378773 6144 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:44.378907 6144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:44.379221 6144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:44.381994 6144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/data/instance:
uuid: "c29bd25a3dd440feb3fc55f28a30bc3e"
format_stamp: "Formatted at 2026-08-07 03:13:44 on dist-test-slave-tm1g"
server_key: "2b407821903ecd1474233f829c0f224e"
server_key_iv: "86256c7fa4d416fd70bf28b37cc74b3e"
server_key_version: "encryptionkey@0"
I20260807 03:13:44.382817 6144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance:
uuid: "c29bd25a3dd440feb3fc55f28a30bc3e"
format_stamp: "Formatted at 2026-08-07 03:13:44 on dist-test-slave-tm1g"
server_key: "2b407821903ecd1474233f829c0f224e"
server_key_iv: "86256c7fa4d416fd70bf28b37cc74b3e"
server_key_version: "encryptionkey@0"
I20260807 03:13:44.387633 6144 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260807 03:13:44.391111 6158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:44.392335 6144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:13:44.392511 6144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/wal
uuid: "c29bd25a3dd440feb3fc55f28a30bc3e"
format_stamp: "Formatted at 2026-08-07 03:13:44 on dist-test-slave-tm1g"
server_key: "2b407821903ecd1474233f829c0f224e"
server_key_iv: "86256c7fa4d416fd70bf28b37cc74b3e"
server_key_version: "encryptionkey@0"
I20260807 03:13:44.392728 6144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:44.413196 6144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:44.413962 6144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:44.414239 6144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:44.415242 6144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:44.416666 6144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:44.416787 6144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:44.416923 6144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:44.417016 6144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:44.450258 6144 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.2:40445
I20260807 03:13:44.450356 6275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.2:40445 every 8 connection(s)
I20260807 03:13:44.451800 6144 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/data/info.pb
I20260807 03:13:44.459280 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 6144
I20260807 03:13:44.459465 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/wal/instance
I20260807 03:13:44.459925 17532 external_mini_cluster.cc:1428] Setting key 016a520bba14e73e5e0915a8b6250864
I20260807 03:13:44.462827 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.31.3:0
--local_ip_for_outbound_sockets=127.17.31.3
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37215
--builtin_ntp_servers=127.17.31.20:43819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 03:13:44.464607 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:44.454490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.2:59713 (local address 127.17.31.62:37215)
0807 03:13:44.454700 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:44.454707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:44.456116 (+ 1409us) server_negotiation.cc:407] Connection header received
0807 03:13:44.457823 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:44.457827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:44.457890 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:44.457988 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:44.459064 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.459938 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.461634 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.462185 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.462331 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:44.463644 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:44.463653 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:44.463662 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:44.463686 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:44.463729 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:44.463748 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:44.463843 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:44.464077 (+ 234us) server_negotiation.cc:299] Negotiation successful
0807 03:13:44.464189 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 03:13:44.465251 6278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:44.454919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:37215 (local address 127.17.31.2:59713)
0807 03:13:44.455939 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:44.455991 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:44.457578 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:44.458183 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:44.458191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:44.458271 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:44.458881 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:44.458919 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.460090 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.460097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:44.461470 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:44.461510 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.462448 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.462453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:44.462647 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:44.463336 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:13:44.463338 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:44.463374 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:13:44.463452 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:44.463855 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:44.463863 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:44.463868 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 03:13:44.463998 (+ 130us) client_negotiation.cc:240] Negotiation successful
0807 03:13:44.464259 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":341,"threads_started":1}
I20260807 03:13:44.466724 6276 heartbeater.cc:347] Connected to a master server at 127.17.31.62:37215
I20260807 03:13:44.467011 6276 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:44.467728 6276 heartbeater.cc:511] Master 127.17.31.62:37215 requested a full tablet report, sending...
I20260807 03:13:44.469080 5947 ts_manager.cc:194] Registered new tserver with Master: c29bd25a3dd440feb3fc55f28a30bc3e (127.17.31.2:40445)
I20260807 03:13:44.469866 5947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.31.2:59713
W20260807 03:13:44.797526 6279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:44.797828 6279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:44.797940 6279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:44.805006 6279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:44.805095 6279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:44.805253 6279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.31.3
I20260807 03:13:44.814079 6279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:43819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.31.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.31.3
--webserver_port=0
--tserver_master_addrs=127.17.31.62:37215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.31.3
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:44.815372 6279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:44.817116 6279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:44.830686 6284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:44.830816 6285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:44.830651 6287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:44.831736 6279 server_base.cc:1034] running on GCE node
I20260807 03:13:44.832283 6279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:44.833184 6279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:44.834400 6279 hybrid_clock.cc:648] HybridClock initialized: now 1786072424834360 us; error 41 us; skew 500 ppm
I20260807 03:13:44.834748 6279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 03:13:44.836956 6279 webserver.cc:460] Webserver started at http://127.17.31.3:43791/ using document root <none> and password file <none>
I20260807 03:13:44.837628 6279 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:44.837749 6279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:44.838075 6279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:44.840937 6279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/data/instance:
uuid: "d504e57c25074914bb481535fc5a8680"
format_stamp: "Formatted at 2026-08-07 03:13:44 on dist-test-slave-tm1g"
server_key: "a3755320be3d0a36b9dac9d78d660cc2"
server_key_iv: "fccd1af279607b8d51c41f1bbbcd5850"
server_key_version: "encryptionkey@0"
I20260807 03:13:44.841713 6279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance:
uuid: "d504e57c25074914bb481535fc5a8680"
format_stamp: "Formatted at 2026-08-07 03:13:44 on dist-test-slave-tm1g"
server_key: "a3755320be3d0a36b9dac9d78d660cc2"
server_key_iv: "fccd1af279607b8d51c41f1bbbcd5850"
server_key_version: "encryptionkey@0"
I20260807 03:13:44.846628 6279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260807 03:13:44.849812 6294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:44.851012 6279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 03:13:44.851172 6279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/wal
uuid: "d504e57c25074914bb481535fc5a8680"
format_stamp: "Formatted at 2026-08-07 03:13:44 on dist-test-slave-tm1g"
server_key: "a3755320be3d0a36b9dac9d78d660cc2"
server_key_iv: "fccd1af279607b8d51c41f1bbbcd5850"
server_key_version: "encryptionkey@0"
I20260807 03:13:44.851377 6279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:44.861433 6279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:44.862143 6279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:44.862473 6279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:44.863330 6279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 03:13:44.864614 6279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 03:13:44.864730 6279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:44.864859 6279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 03:13:44.864946 6279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:44.897348 6279 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.3:37999
I20260807 03:13:44.897393 6408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.3:37999 every 8 connection(s)
I20260807 03:13:44.898742 6279 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/data/info.pb
I20260807 03:13:44.902068 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 6279
I20260807 03:13:44.902242 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/wal/instance
I20260807 03:13:44.902724 17532 external_mini_cluster.cc:1428] Setting key 895f790a9417201c93f0e3fda74c26e8
I20260807 03:13:44.912284 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:44.901551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.3:43217 (local address 127.17.31.62:37215)
0807 03:13:44.901771 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:44.901778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:44.903341 (+ 1563us) server_negotiation.cc:407] Connection header received
0807 03:13:44.905706 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:44.905710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:44.905775 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:44.905896 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:44.907107 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.907860 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.909402 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.909849 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.909991 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:44.911462 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:44.911467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:44.911470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:44.911488 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:44.911518 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:44.911543 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:44.911639 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:44.911871 (+ 232us) server_negotiation.cc:299] Negotiation successful
0807 03:13:44.911966 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 03:13:44.913260 6411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:44.902053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:37215 (local address 127.17.31.3:43217)
0807 03:13:44.903165 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:44.903221 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:44.905098 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:44.906115 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:44.906123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:44.906210 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:13:44.906945 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:44.906962 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.908022 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.908026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:44.909234 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:44.909246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.910078 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.910085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:44.910405 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:44.911116 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 03:13:44.911119 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:13:44.911160 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 03:13:44.911228 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:13:44.911639 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:13:44.911653 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:13:44.911662 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 03:13:44.911839 (+ 177us) client_negotiation.cc:240] Negotiation successful
0807 03:13:44.912182 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":394,"threads_started":1}
I20260807 03:13:44.914474 6409 heartbeater.cc:347] Connected to a master server at 127.17.31.62:37215
I20260807 03:13:44.914745 6409 heartbeater.cc:464] Registering TS with master...
I20260807 03:13:44.915448 6409 heartbeater.cc:511] Master 127.17.31.62:37215 requested a full tablet report, sending...
I20260807 03:13:44.916747 5947 ts_manager.cc:194] Registered new tserver with Master: d504e57c25074914bb481535fc5a8680 (127.17.31.3:37999)
I20260807 03:13:44.917665 5947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.31.3:43217
I20260807 03:13:44.917698 17532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 03:13:44.927016 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:44.921888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47422 (local address 127.17.31.62:37215)
0807 03:13:44.922051 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:44.922055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:44.922547 (+ 492us) server_negotiation.cc:407] Connection header received
0807 03:13:44.922668 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:44.922672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:44.922735 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:44.922810 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:44.923233 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.923879 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.925038 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.925461 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.925551 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:44.926520 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:44.926524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:44.926527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:44.926542 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:44.926570 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:44.926581 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:44.926648 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:44.926758 (+ 110us) server_negotiation.cc:299] Negotiation successful
0807 03:13:44.926821 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 03:13:44.930321 5947 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 03:13:44.937392 5947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 03:13:44.969131 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:44.949009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.17.31.3:37999)
0807 03:13:44.960735 (+ 11726us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:44.960745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:44.960786 (+ 41us) server_negotiation.cc:407] Connection header received
0807 03:13:44.960886 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:44.960894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:44.961117 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:44.961306 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:44.962063 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.963717 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.967079 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.968000 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.968161 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:44.968460 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:44.968614 (+ 154us) server_negotiation.cc:299] Negotiation successful
0807 03:13:44.968779 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11517,"thread_start_us":11332,"threads_started":1}
I20260807 03:13:44.969168 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:44.948712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:37999 (local address 127.0.0.1:52238)
0807 03:13:44.959174 (+ 10462us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:44.959215 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:44.959395 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:44.961314 (+ 1919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:44.961324 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:44.961367 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:44.961785 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:44.961798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.963871 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.963877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:44.966912 (+ 3035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:44.966922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.968190 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.968196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:44.968389 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:44.968450 (+ 61us) client_negotiation.cc:769] Sending connection context
0807 03:13:44.968578 (+ 128us) client_negotiation.cc:240] Negotiation successful
0807 03:13:44.968771 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10216,"mutex_wait_us":34,"thread_start_us":10977,"threads_started":1}
I20260807 03:13:44.969738 6425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:44.950214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:40445 (local address 127.0.0.1:40352)
0807 03:13:44.960334 (+ 10120us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:44.960355 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:44.960465 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:44.961643 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:44.961650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:44.961699 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:44.962580 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:44.962594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.964304 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.964311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:44.966636 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:44.966653 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.968738 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.968746 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:44.969022 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:44.969134 (+ 112us) client_negotiation.cc:769] Sending connection context
0807 03:13:44.969206 (+ 72us) client_negotiation.cc:240] Negotiation successful
0807 03:13:44.969331 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9981,"mutex_wait_us":144,"spinlock_wait_cycles":4352,"thread_start_us":9853,"threads_started":1}
I20260807 03:13:44.969810 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:44.949060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.17.31.1:44799)
0807 03:13:44.958399 (+ 9339us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:44.958412 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:44.960651 (+ 2239us) server_negotiation.cc:407] Connection header received
0807 03:13:44.960961 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:44.960970 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:44.961191 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:44.961351 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:44.962845 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.964132 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.966891 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.968245 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.968353 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:44.968584 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:44.969299 (+ 715us) server_negotiation.cc:299] Negotiation successful
0807 03:13:44.969472 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9094,"thread_start_us":9349,"threads_started":1}
I20260807 03:13:44.970489 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:44.951086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.17.31.2:40445)
0807 03:13:44.961092 (+ 10006us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:44.961100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:44.961124 (+ 24us) server_negotiation.cc:407] Connection header received
0807 03:13:44.961210 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:44.961220 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:44.961417 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:44.961695 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:44.962799 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.964135 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.966805 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.968243 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.968370 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:44.968584 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:44.969991 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0807 03:13:44.970152 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9850,"thread_start_us":9715,"threads_started":1}
I20260807 03:13:44.970635 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:44.948828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.1:44799 (local address 127.0.0.1:52450)
0807 03:13:44.959944 (+ 11116us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:44.960575 (+ 631us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:44.960748 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:44.961640 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:44.961644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:44.961680 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:44.962633 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:44.962648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.964249 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.964256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:44.966317 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:44.966331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:44.968754 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:44.968759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:44.969018 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:44.969126 (+ 108us) client_negotiation.cc:769] Sending connection context
0807 03:13:44.970127 (+ 1001us) client_negotiation.cc:240] Negotiation successful
0807 03:13:44.970242 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10990,"mutex_wait_us":344,"spinlock_wait_cycles":768,"thread_start_us":9926,"threads_started":1}
I20260807 03:13:44.973268 6208 tablet_service.cc:1467] Processing CreateTablet for tablet 81777335f82d48a48f24beeeca880af4 (DEFAULT_TABLE table=test-table [id=04176f5fa2df40379c7cc26fcea37d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:44.973215 6343 tablet_service.cc:1467] Processing CreateTablet for tablet 81777335f82d48a48f24beeeca880af4 (DEFAULT_TABLE table=test-table [id=04176f5fa2df40379c7cc26fcea37d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:44.974349 6208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81777335f82d48a48f24beeeca880af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:44.974300 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81777335f82d48a48f24beeeca880af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:44.977641 6070 tablet_service.cc:1467] Processing CreateTablet for tablet 81777335f82d48a48f24beeeca880af4 (DEFAULT_TABLE table=test-table [id=04176f5fa2df40379c7cc26fcea37d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 03:13:44.978962 6070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81777335f82d48a48f24beeeca880af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:44.985961 6428 tablet_bootstrap.cc:492] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e: Bootstrap starting.
I20260807 03:13:44.989295 6429 tablet_bootstrap.cc:492] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680: Bootstrap starting.
I20260807 03:13:44.989828 6428 tablet_bootstrap.cc:654] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:44.991216 6428 log.cc:824] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:44.992642 6429 tablet_bootstrap.cc:654] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:44.993829 6429 log.cc:824] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:44.995419 6428 tablet_bootstrap.cc:492] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e: No bootstrap required, opened a new log
I20260807 03:13:44.996292 6428 ts_tablet_manager.cc:1397] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260807 03:13:45.003382 6432 tablet_bootstrap.cc:492] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388: Bootstrap starting.
I20260807 03:13:45.005323 6429 tablet_bootstrap.cc:492] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680: No bootstrap required, opened a new log
I20260807 03:13:45.005594 6429 ts_tablet_manager.cc:1397] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.008s
I20260807 03:13:45.007019 6432 tablet_bootstrap.cc:654] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:45.008252 6432 log.cc:824] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:45.009248 6428 raft_consensus.cc:348] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d504e57c25074914bb481535fc5a8680" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 37999 } } peers { permanent_uuid: "4ad0975eff90445493314f9a660d1388" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44799 } } peers { permanent_uuid: "c29bd25a3dd440feb3fc55f28a30bc3e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40445 } }
I20260807 03:13:45.009662 6428 raft_consensus.cc:374] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:45.009795 6428 raft_consensus.cc:729] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c29bd25a3dd440feb3fc55f28a30bc3e, State: Initialized, Role: FOLLOWER
I20260807 03:13:45.010416 6428 consensus_queue.cc:260] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d504e57c25074914bb481535fc5a8680" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 37999 } } peers { permanent_uuid: "4ad0975eff90445493314f9a660d1388" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44799 } } peers { permanent_uuid: "c29bd25a3dd440feb3fc55f28a30bc3e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40445 } }
I20260807 03:13:45.011566 6276 heartbeater.cc:503] Master 127.17.31.62:37215 was elected leader, sending a full tablet report...
I20260807 03:13:45.012176 6428 ts_tablet_manager.cc:1428] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20260807 03:13:45.011332 6432 tablet_bootstrap.cc:492] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388: No bootstrap required, opened a new log
I20260807 03:13:45.012665 6432 ts_tablet_manager.cc:1397] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260807 03:13:45.013854 6139 heartbeater.cc:503] Master 127.17.31.62:37215 was elected leader, sending a full tablet report...
I20260807 03:13:45.014664 6429 raft_consensus.cc:348] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d504e57c25074914bb481535fc5a8680" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 37999 } } peers { permanent_uuid: "4ad0975eff90445493314f9a660d1388" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44799 } } peers { permanent_uuid: "c29bd25a3dd440feb3fc55f28a30bc3e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40445 } }
I20260807 03:13:45.015089 6429 raft_consensus.cc:374] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:45.015185 6429 raft_consensus.cc:729] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d504e57c25074914bb481535fc5a8680, State: Initialized, Role: FOLLOWER
I20260807 03:13:45.015810 6429 consensus_queue.cc:260] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d504e57c25074914bb481535fc5a8680" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 37999 } } peers { permanent_uuid: "4ad0975eff90445493314f9a660d1388" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44799 } } peers { permanent_uuid: "c29bd25a3dd440feb3fc55f28a30bc3e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40445 } }
I20260807 03:13:45.017372 6409 heartbeater.cc:503] Master 127.17.31.62:37215 was elected leader, sending a full tablet report...
I20260807 03:13:45.018124 6429 ts_tablet_manager.cc:1428] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680: Time spent starting tablet: real 0.012s user 0.006s sys 0.007s
I20260807 03:13:45.022168 6432 raft_consensus.cc:348] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d504e57c25074914bb481535fc5a8680" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 37999 } } peers { permanent_uuid: "4ad0975eff90445493314f9a660d1388" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44799 } } peers { permanent_uuid: "c29bd25a3dd440feb3fc55f28a30bc3e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40445 } }
I20260807 03:13:45.022553 6432 raft_consensus.cc:374] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:45.022626 6432 raft_consensus.cc:729] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad0975eff90445493314f9a660d1388, State: Initialized, Role: FOLLOWER
I20260807 03:13:45.023087 6432 consensus_queue.cc:260] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d504e57c25074914bb481535fc5a8680" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 37999 } } peers { permanent_uuid: "4ad0975eff90445493314f9a660d1388" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44799 } } peers { permanent_uuid: "c29bd25a3dd440feb3fc55f28a30bc3e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40445 } }
I20260807 03:13:45.024497 6432 ts_tablet_manager.cc:1428] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
W20260807 03:13:45.150864 6410 tablet.cc:2369] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:13:45.150864 6436 raft_consensus.cc:482] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 03:13:45.151243 6436 raft_consensus.cc:504] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d504e57c25074914bb481535fc5a8680" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 37999 } } peers { permanent_uuid: "4ad0975eff90445493314f9a660d1388" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44799 } } peers { permanent_uuid: "c29bd25a3dd440feb3fc55f28a30bc3e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40445 } }
I20260807 03:13:45.153129 6436 leader_election.cc:288] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d504e57c25074914bb481535fc5a8680 (127.17.31.3:37999), c29bd25a3dd440feb3fc55f28a30bc3e (127.17.31.2:40445)
I20260807 03:13:45.160034 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:45.153616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:54137 (local address 127.17.31.3:37999)
0807 03:13:45.153958 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:45.153967 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:45.154407 (+ 440us) server_negotiation.cc:407] Connection header received
0807 03:13:45.154744 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:45.154748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:45.154861 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:45.154963 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:45.155669 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:45.156461 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:45.158384 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:45.159134 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:45.159275 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:45.159372 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:45.159676 (+ 304us) server_negotiation.cc:299] Negotiation successful
0807 03:13:45.159799 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260807 03:13:45.160097 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:45.153671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.31.1:38089 (local address 127.17.31.2:40445)
0807 03:13:45.154165 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:45.154173 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:45.154709 (+ 536us) server_negotiation.cc:407] Connection header received
0807 03:13:45.154918 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:45.154926 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:45.155019 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:45.155118 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 03:13:45.155899 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:45.156584 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:45.158435 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:45.159181 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:45.159314 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:45.159422 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:45.159726 (+ 304us) server_negotiation.cc:299] Negotiation successful
0807 03:13:45.159850 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314}
I20260807 03:13:45.160168 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:45.153415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.3:37999 (local address 127.17.31.1:54137)
0807 03:13:45.154257 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:45.154278 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:45.154491 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:45.154988 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:45.154991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:45.155012 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:45.155510 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:45.155528 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:45.156587 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:45.156593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:45.158167 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:45.158180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:45.159328 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:45.159334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:45.159499 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:45.159527 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 03:13:45.159627 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 03:13:45.159762 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":73,"thread_start_us":457,"threads_started":1}
I20260807 03:13:45.160555 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:45.153566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.2:40445 (local address 127.17.31.1:38089)
0807 03:13:45.154593 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0807 03:13:45.154610 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:13:45.154770 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:13:45.155149 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:13:45.155155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:13:45.155191 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 03:13:45.155705 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:45.155722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:45.156755 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:45.156760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:45.158206 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:13:45.158222 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:13:45.159380 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:13:45.159386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:13:45.159562 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:45.159585 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 03:13:45.159687 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 03:13:45.159821 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":7,"thread_start_us":633,"threads_started":1}
I20260807 03:13:45.161238 6363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81777335f82d48a48f24beeeca880af4" candidate_uuid: "4ad0975eff90445493314f9a660d1388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d504e57c25074914bb481535fc5a8680" is_pre_election: true
I20260807 03:13:45.161533 6229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81777335f82d48a48f24beeeca880af4" candidate_uuid: "4ad0975eff90445493314f9a660d1388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c29bd25a3dd440feb3fc55f28a30bc3e" is_pre_election: true
I20260807 03:13:45.161835 6363 raft_consensus.cc:2454] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ad0975eff90445493314f9a660d1388 in term 0.
I20260807 03:13:45.162076 6229 raft_consensus.cc:2454] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ad0975eff90445493314f9a660d1388 in term 0.
I20260807 03:13:45.162616 6026 leader_election.cc:302] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ad0975eff90445493314f9a660d1388, d504e57c25074914bb481535fc5a8680; no voters:
I20260807 03:13:45.163024 6436 raft_consensus.cc:2781] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 03:13:45.163139 6436 raft_consensus.cc:482] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 03:13:45.163280 6436 raft_consensus.cc:3037] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:45.165441 6436 raft_consensus.cc:504] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d504e57c25074914bb481535fc5a8680" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 37999 } } peers { permanent_uuid: "4ad0975eff90445493314f9a660d1388" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44799 } } peers { permanent_uuid: "c29bd25a3dd440feb3fc55f28a30bc3e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40445 } }
I20260807 03:13:45.166031 6436 leader_election.cc:288] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [CANDIDATE]: Term 1 election: Requested vote from peers d504e57c25074914bb481535fc5a8680 (127.17.31.3:37999), c29bd25a3dd440feb3fc55f28a30bc3e (127.17.31.2:40445)
I20260807 03:13:45.166460 6363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81777335f82d48a48f24beeeca880af4" candidate_uuid: "4ad0975eff90445493314f9a660d1388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d504e57c25074914bb481535fc5a8680"
I20260807 03:13:45.166636 6363 raft_consensus.cc:3037] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:45.166643 6229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81777335f82d48a48f24beeeca880af4" candidate_uuid: "4ad0975eff90445493314f9a660d1388" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c29bd25a3dd440feb3fc55f28a30bc3e"
I20260807 03:13:45.166867 6229 raft_consensus.cc:3037] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:45.168790 6363 raft_consensus.cc:2454] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ad0975eff90445493314f9a660d1388 in term 1.
I20260807 03:13:45.169164 6026 leader_election.cc:302] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ad0975eff90445493314f9a660d1388, d504e57c25074914bb481535fc5a8680; no voters:
I20260807 03:13:45.169229 6229 raft_consensus.cc:2454] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ad0975eff90445493314f9a660d1388 in term 1.
I20260807 03:13:45.169463 6436 raft_consensus.cc:2781] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:45.169925 6436 raft_consensus.cc:686] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [term 1 LEADER]: Becoming Leader. State: Replica: 4ad0975eff90445493314f9a660d1388, State: Running, Role: LEADER
I20260807 03:13:45.170495 6436 consensus_queue.cc:237] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d504e57c25074914bb481535fc5a8680" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 37999 } } peers { permanent_uuid: "4ad0975eff90445493314f9a660d1388" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44799 } } peers { permanent_uuid: "c29bd25a3dd440feb3fc55f28a30bc3e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40445 } }
I20260807 03:13:45.174798 5946 catalog_manager.cc:5495] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ad0975eff90445493314f9a660d1388 (127.17.31.1). New cstate: current_term: 1 leader_uuid: "4ad0975eff90445493314f9a660d1388" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d504e57c25074914bb481535fc5a8680" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 37999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ad0975eff90445493314f9a660d1388" member_type: VOTER last_known_addr { host: "127.17.31.1" port: 44799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c29bd25a3dd440feb3fc55f28a30bc3e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40445 } health_report { overall_health: UNKNOWN } } }
I20260807 03:13:45.203450 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:45.198345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52452 (local address 127.17.31.1:44799)
0807 03:13:45.198553 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:45.198562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:45.198632 (+ 70us) server_negotiation.cc:407] Connection header received
0807 03:13:45.198817 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:45.198821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:45.198908 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:45.199028 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:13:45.199332 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:45.200013 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:45.201396 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:45.201805 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:45.201908 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:45.202336 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:13:45.202874 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:13:45.202994 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:45.203063 (+ 69us) server_negotiation.cc:299] Negotiation successful
0807 03:13:45.203124 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260807 03:13:45.203842 6277 tablet.cc:2369] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:13:45.211181 6363 raft_consensus.cc:1261] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680 [term 1 FOLLOWER]: Refusing update from remote peer 4ad0975eff90445493314f9a660d1388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:13:45.211181 6229 raft_consensus.cc:1261] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e [term 1 FOLLOWER]: Refusing update from remote peer 4ad0975eff90445493314f9a660d1388: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:13:45.212116 6439 consensus_queue.cc:1035] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c29bd25a3dd440feb3fc55f28a30bc3e" member_type: VOTER last_known_addr { host: "127.17.31.2" port: 40445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:13:45.212457 6436 consensus_queue.cc:1035] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d504e57c25074914bb481535fc5a8680" member_type: VOTER last_known_addr { host: "127.17.31.3" port: 37999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:13:45.223646 6442 mvcc.cc:205] Tried to move back new op lower bound from 7315752653655732224 to 7315752653502103552. Current Snapshot: MvccSnapshot[applied={T|T < 7315752653655732224}]
I20260807 03:13:45.239305 5947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:13:45.239562 5947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 03:13:45.243052 5947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7305b13923a24d1eb90932030bedd699: Sending DeleteTablet for 3 replicas of tablet 81777335f82d48a48f24beeeca880af4
I20260807 03:13:45.244222 6070 tablet_service.cc:1514] Processing DeleteTablet for tablet 81777335f82d48a48f24beeeca880af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:45 UTC) from {username='slave'} at 127.0.0.1:52450
I20260807 03:13:45.244323 6343 tablet_service.cc:1514] Processing DeleteTablet for tablet 81777335f82d48a48f24beeeca880af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:45 UTC) from {username='slave'} at 127.0.0.1:52238
I20260807 03:13:45.244352 6208 tablet_service.cc:1514] Processing DeleteTablet for tablet 81777335f82d48a48f24beeeca880af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 03:13:45 UTC) from {username='slave'} at 127.0.0.1:40352
I20260807 03:13:45.251494 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 6006
I20260807 03:13:45.257853 6452 tablet_replica.cc:330] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680: stopping tablet replica
I20260807 03:13:45.258433 6452 raft_consensus.cc:2229] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:13:45.259044 6452 raft_consensus.cc:2258] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:45.259835 6453 tablet_replica.cc:330] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388: stopping tablet replica
I20260807 03:13:45.260175 6453 raft_consensus.cc:2229] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [term 1 LEADER]: Raft consensus shutting down.
I20260807 03:13:45.260767 6453 raft_consensus.cc:2258] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:45.261716 6454 tablet_replica.cc:330] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e: stopping tablet replica
I20260807 03:13:45.262040 6454 raft_consensus.cc:2229] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 03:13:45.262485 6454 raft_consensus.cc:2258] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 03:13:45.264024 6454 ts_tablet_manager.cc:1905] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:45.264463 6452 ts_tablet_manager.cc:1905] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:45.268781 6454 ts_tablet_manager.cc:1918] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:45.268958 6454 log.cc:1196] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-1/wal/wals/81777335f82d48a48f24beeeca880af4
I20260807 03:13:45.269637 6454 ts_tablet_manager.cc:1939] T 81777335f82d48a48f24beeeca880af4 P c29bd25a3dd440feb3fc55f28a30bc3e: Deleting consensus metadata
I20260807 03:13:45.269896 6453 ts_tablet_manager.cc:1905] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 03:13:45.271579 5933 catalog_manager.cc:4841] TS c29bd25a3dd440feb3fc55f28a30bc3e (127.17.31.2:40445): tablet 81777335f82d48a48f24beeeca880af4 (table test-table [id=04176f5fa2df40379c7cc26fcea37d7d]) successfully deleted
I20260807 03:13:45.271962 6452 ts_tablet_manager.cc:1918] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:45.272117 6452 log.cc:1196] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-2/wal/wals/81777335f82d48a48f24beeeca880af4
I20260807 03:13:45.272691 6452 ts_tablet_manager.cc:1939] T 81777335f82d48a48f24beeeca880af4 P d504e57c25074914bb481535fc5a8680: Deleting consensus metadata
I20260807 03:13:45.274271 5934 catalog_manager.cc:4841] TS d504e57c25074914bb481535fc5a8680 (127.17.31.3:37999): tablet 81777335f82d48a48f24beeeca880af4 (table test-table [id=04176f5fa2df40379c7cc26fcea37d7d]) successfully deleted
I20260807 03:13:45.276674 6453 ts_tablet_manager.cc:1918] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 03:13:45.276902 6453 log.cc:1196] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388: Deleting WAL directory at /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786072106928531-17532-0/minicluster-data/ts-0/wal/wals/81777335f82d48a48f24beeeca880af4
I20260807 03:13:45.277514 6453 ts_tablet_manager.cc:1939] T 81777335f82d48a48f24beeeca880af4 P 4ad0975eff90445493314f9a660d1388: Deleting consensus metadata
I20260807 03:13:45.277985 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:45.252779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.17.31.1:44799)
0807 03:13:45.253012 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:45.253021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:45.269400 (+ 16379us) server_negotiation.cc:407] Connection header received
0807 03:13:45.269835 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:45.269842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:45.269952 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:45.270025 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:45.270772 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:45.271992 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:45.273936 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:45.274708 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:45.274851 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:45.276123 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:45.276133 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:45.276149 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:45.277079 (+ 930us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:45.277159 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:45.277191 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:45.277312 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:45.277482 (+ 170us) server_negotiation.cc:299] Negotiation successful
0807 03:13:45.277613 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 03:13:45.278788 6134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:45.279196 5931 catalog_manager.cc:4841] TS 4ad0975eff90445493314f9a660d1388 (127.17.31.1:44799): tablet 81777335f82d48a48f24beeeca880af4 (table test-table [id=04176f5fa2df40379c7cc26fcea37d7d]) successfully deleted
I20260807 03:13:45.374464 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 6006
I20260807 03:13:45.393786 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 6144
I20260807 03:13:45.407258 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:45.394788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.17.31.2:40445)
0807 03:13:45.394991 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:45.394999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:45.395020 (+ 21us) server_negotiation.cc:407] Connection header received
0807 03:13:45.395092 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:45.395096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:45.395172 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:45.395296 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:45.395749 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:45.396610 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:45.397932 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:45.398486 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:45.398593 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:45.405684 (+ 7091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:45.405693 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:45.405709 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:45.406491 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:45.406542 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:45.406564 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:45.406659 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:45.406790 (+ 131us) server_negotiation.cc:299] Negotiation successful
0807 03:13:45.406883 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 03:13:45.407936 6271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:45.509825 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 6144
I20260807 03:13:45.530128 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 6279
I20260807 03:13:45.536597 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:45.530950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52246 (local address 127.17.31.3:37999)
0807 03:13:45.531126 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:45.531131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:45.531149 (+ 18us) server_negotiation.cc:407] Connection header received
0807 03:13:45.531300 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:45.531304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:45.531390 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:45.531483 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:45.531855 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:45.532551 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:45.533777 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:45.534151 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:45.534238 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:45.535308 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:45.535316 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:45.535333 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:45.535904 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:45.535949 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:45.535967 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:45.536057 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:45.536172 (+ 115us) server_negotiation.cc:299] Negotiation successful
0807 03:13:45.536254 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:13:45.537226 6404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:45.624897 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 6279
I20260807 03:13:45.643942 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 5915
I20260807 03:13:45.645021 5978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:45.756662 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 5915
2026-08-07T03:13:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2818 ms)
[----------] 6 tests from EncryptionPolicyTest (21347 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:13:45 dist-test-slave-tm1g krb5kdc[6462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:13:45 dist-test-slave-tm1g krb5kdc[6462](info): set up 2 sockets
Aug 07 03:13:45 dist-test-slave-tm1g krb5kdc[6462](info): commencing operation
krb5kdc: starting...
W20260807 03:13:46.896188 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.008s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:13:47 dist-test-slave-tm1g krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:13:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:13:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:13:47.171912 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42123
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:47.511842 6476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:47.512176 6476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:47.512354 6476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:47.519321 6476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:13:47.519464 6476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:47.519570 6476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:47.519683 6476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:13:47.519773 6476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:13:47.529201 6476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42123
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:47.530486 6476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:47.531765 6476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:47.543674 6481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:47.543787 6484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:47.543632 6482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:47.544692 6476 server_base.cc:1034] running on GCE node
I20260807 03:13:47.545430 6476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:47.546856 6476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:47.548086 6476 hybrid_clock.cc:648] HybridClock initialized: now 1786072427548049 us; error 45 us; skew 500 ppm
I20260807 03:13:47.548416 6476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:47 dist-test-slave-tm1g krb5kdc[6462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:47.551432 6476 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:13:47.552842 6476 webserver.cc:460] Webserver started at http://127.0.0.1:37141/ using document root <none> and password file <none>
I20260807 03:13:47.553507 6476 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:47.553653 6476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:47.553947 6476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:47.557061 6476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "ebf17733d96d4169bc102b50c355e2ba"
format_stamp: "Formatted at 2026-08-07 03:13:47 on dist-test-slave-tm1g"
server_key: "4586378e7647da33924e4cc053724de4"
server_key_iv: "df1e14f65c4fcf72548a48d90043ce3e"
server_key_version: "encryptionkey@0"
I20260807 03:13:47.557832 6476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "ebf17733d96d4169bc102b50c355e2ba"
format_stamp: "Formatted at 2026-08-07 03:13:47 on dist-test-slave-tm1g"
server_key: "4586378e7647da33924e4cc053724de4"
server_key_iv: "df1e14f65c4fcf72548a48d90043ce3e"
server_key_version: "encryptionkey@0"
I20260807 03:13:47.562721 6476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260807 03:13:47.566002 6492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:47.567291 6476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:13:47.567488 6476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "ebf17733d96d4169bc102b50c355e2ba"
format_stamp: "Formatted at 2026-08-07 03:13:47 on dist-test-slave-tm1g"
server_key: "4586378e7647da33924e4cc053724de4"
server_key_iv: "df1e14f65c4fcf72548a48d90043ce3e"
server_key_version: "encryptionkey@0"
I20260807 03:13:47.567679 6476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:47.588066 6476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:47.591858 6476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:47.592133 6476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:47.610551 6476 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42123
I20260807 03:13:47.610594 6544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42123 every 8 connection(s)
I20260807 03:13:47.611863 6476 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:13:47.615474 6545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:47.617522 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 6476
I20260807 03:13:47.617770 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:13:47.618613 17532 external_mini_cluster.cc:1428] Setting key 6fac1da45c6df019b86466ea795867ce
I20260807 03:13:47.628177 6545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba: Bootstrap starting.
I20260807 03:13:47.632078 6545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:47.633323 6545 log.cc:824] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba: Log is configured to *not* fsync() on all Append() calls
Aug 07 03:13:47 dist-test-slave-tm1g krb5kdc[6462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 03:13:47.636161 6545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba: No bootstrap required, opened a new log
I20260807 03:13:47.643839 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:47.620556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56343 (local address 127.0.0.1:42123)
0807 03:13:47.621348 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:47.621362 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:47.621402 (+ 40us) server_negotiation.cc:407] Connection header received
0807 03:13:47.623072 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:47.623093 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:47.623459 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:47.623757 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:47.625576 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:47.626943 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:47.628823 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:47.629441 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:47.629574 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:47.634619 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:47.634626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:47.634639 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:47.634676 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:47.638363 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:47.639441 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:47.639448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:47.639455 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:47.639687 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:47.640426 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:47.640432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:47.640436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:47.641244 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:47.641562 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:47.642117 (+ 555us) server_negotiation.cc:299] Negotiation successful
0807 03:13:47.642489 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":374,"threads_started":1}
I20260807 03:13:47.647089 6545 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf17733d96d4169bc102b50c355e2ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42123 } }
I20260807 03:13:47.647501 6545 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:47.647612 6545 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebf17733d96d4169bc102b50c355e2ba, State: Initialized, Role: FOLLOWER
I20260807 03:13:47.648274 6545 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf17733d96d4169bc102b50c355e2ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42123 } }
I20260807 03:13:47.648579 6545 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:13:47.648732 6545 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:13:47.648916 6545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:47.649035 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:13:47.652263 6545 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf17733d96d4169bc102b50c355e2ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42123 } }
I20260807 03:13:47.652882 6545 leader_election.cc:302] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebf17733d96d4169bc102b50c355e2ba; no voters:
I20260807 03:13:47.653602 6545 leader_election.cc:288] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:13:47.653707 6552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:47.655961 6545 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:13:47.656348 6552 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [term 1 LEADER]: Becoming Leader. State: Replica: ebf17733d96d4169bc102b50c355e2ba, State: Running, Role: LEADER
I20260807 03:13:47.656931 6552 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf17733d96d4169bc102b50c355e2ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42123 } }
I20260807 03:13:47.660089 6556 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebf17733d96d4169bc102b50c355e2ba. Latest consensus state: current_term: 1 leader_uuid: "ebf17733d96d4169bc102b50c355e2ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf17733d96d4169bc102b50c355e2ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42123 } } }
I20260807 03:13:47.660128 6555 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebf17733d96d4169bc102b50c355e2ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebf17733d96d4169bc102b50c355e2ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42123 } } }
I20260807 03:13:47.660465 6556 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:47.660511 6555 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:47.661839 6564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:13:47.666697 6564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:13:47.680287 6564 catalog_manager.cc:1347] Generated new cluster ID: 50f646a369a54e50917ba7c995ab09a2
I20260807 03:13:47.680392 6564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:13:47.684031 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:47.655356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58665 (local address 127.0.0.1:42123)
0807 03:13:47.655529 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:47.655534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:47.656270 (+ 736us) server_negotiation.cc:407] Connection header received
0807 03:13:47.656538 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:47.656543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:47.656631 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:47.656702 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:47.661915 (+ 5213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:47.663601 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:47.666016 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:47.667026 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:47.667166 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:47.671106 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:47.671115 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:47.671122 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:47.671176 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:47.676154 (+ 4978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:47.681419 (+ 5265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:47.681435 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:47.681438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:47.681608 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:47.682175 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:47.682180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:47.682184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:47.682594 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:47.682843 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:47.683520 (+ 677us) server_negotiation.cc:299] Negotiation successful
0807 03:13:47.683701 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:13:47.691851 6564 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:13:47.693060 6564 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:13:47.700563 6564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebf17733d96d4169bc102b50c355e2ba: Generated new TSK 0
I20260807 03:13:47.701488 6564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:13:47.709429 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 6476
I20260807 03:13:47.711136 6540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:47.803632 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 6476
W20260807 03:13:47.818858 6460 connection.cc:535] client connection to 127.0.0.1:42123 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42123): Connection reset by peer (error 104)
2026-08-07T03:13:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2035 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:13:47 dist-test-slave-tm1g krb5kdc[6578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:13:47 dist-test-slave-tm1g krb5kdc[6578](info): set up 2 sockets
Aug 07 03:13:47 dist-test-slave-tm1g krb5kdc[6578](info): commencing operation
krb5kdc: starting...
W20260807 03:13:48.923723 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:13:49 dist-test-slave-tm1g krb5kdc[6578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:13:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:13:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:13:49.183099 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35007
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:49.509089 6591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:49.509359 6591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:49.509523 6591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:49.516629 6591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:13:49.516726 6591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:49.516860 6591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:49.516965 6591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:13:49.517050 6591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:13:49.526603 6591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35007
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:49.527843 6591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:49.529127 6591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:49.540715 6596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:49.540767 6597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:49.540747 6599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:49.541553 6591 server_base.cc:1034] running on GCE node
I20260807 03:13:49.542243 6591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:49.543560 6591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:49.544780 6591 hybrid_clock.cc:648] HybridClock initialized: now 1786072429544739 us; error 53 us; skew 500 ppm
I20260807 03:13:49.545100 6591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:49 dist-test-slave-tm1g krb5kdc[6578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:49.548087 6591 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:13:49.549430 6591 webserver.cc:460] Webserver started at http://127.0.0.1:35103/ using document root <none> and password file <none>
I20260807 03:13:49.550076 6591 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:49.550220 6591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:49.550550 6591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:49.553397 6591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "18a1ca4a30954618bcd0b12a23cab3bc"
format_stamp: "Formatted at 2026-08-07 03:13:49 on dist-test-slave-tm1g"
server_key: "c00783aa98cab1d1e96715f5d085305e"
server_key_iv: "f37a3c86b66cd6bc2293eaf33d127273"
server_key_version: "encryptionkey@0"
I20260807 03:13:49.554159 6591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "18a1ca4a30954618bcd0b12a23cab3bc"
format_stamp: "Formatted at 2026-08-07 03:13:49 on dist-test-slave-tm1g"
server_key: "c00783aa98cab1d1e96715f5d085305e"
server_key_iv: "f37a3c86b66cd6bc2293eaf33d127273"
server_key_version: "encryptionkey@0"
I20260807 03:13:49.559067 6591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260807 03:13:49.562402 6606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:49.563611 6591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:13:49.563784 6591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "18a1ca4a30954618bcd0b12a23cab3bc"
format_stamp: "Formatted at 2026-08-07 03:13:49 on dist-test-slave-tm1g"
server_key: "c00783aa98cab1d1e96715f5d085305e"
server_key_iv: "f37a3c86b66cd6bc2293eaf33d127273"
server_key_version: "encryptionkey@0"
I20260807 03:13:49.563974 6591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:49.584045 6591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:49.587702 6591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:49.588027 6591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:49.605875 6591 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35007
I20260807 03:13:49.605953 6658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35007 every 8 connection(s)
I20260807 03:13:49.607178 6591 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:13:49.610687 6659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:49.616276 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 6591
I20260807 03:13:49.616482 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:13:49.617043 17532 external_mini_cluster.cc:1428] Setting key ea2da980b2e09bfbc34d3fdffaaf1a74
I20260807 03:13:49.621246 6659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc: Bootstrap starting.
I20260807 03:13:49.624603 6659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:49.625823 6659 log.cc:824] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:49.628943 6659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc: No bootstrap required, opened a new log
I20260807 03:13:49.630067 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:49.618798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37551 (local address 127.0.0.1:35007)
0807 03:13:49.619596 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:49.619607 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:49.619646 (+ 39us) server_negotiation.cc:407] Connection header received
0807 03:13:49.621200 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:49.621220 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:49.621606 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:49.621901 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:49.622477 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:49.623800 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:49.625749 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:49.626719 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:49.626935 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:49.628027 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:49.628034 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:49.628047 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:49.628536 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:49.628605 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:49.628627 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:49.628736 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:49.629066 (+ 330us) server_negotiation.cc:299] Negotiation successful
0807 03:13:49.629207 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":390,"threads_started":1}
I20260807 03:13:49.633677 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:13:49.636873 6659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a1ca4a30954618bcd0b12a23cab3bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35007 } }
I20260807 03:13:49.637238 6659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:49.637383 6659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a1ca4a30954618bcd0b12a23cab3bc, State: Initialized, Role: FOLLOWER
I20260807 03:13:49.638000 6659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a1ca4a30954618bcd0b12a23cab3bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35007 } }
I20260807 03:13:49.638398 6659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:13:49.638583 6659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:13:49.638790 6659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:49.642838 6659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a1ca4a30954618bcd0b12a23cab3bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35007 } }
I20260807 03:13:49.643332 6659 leader_election.cc:302] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18a1ca4a30954618bcd0b12a23cab3bc; no voters:
I20260807 03:13:49.644054 6659 leader_election.cc:288] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:13:49.644164 6669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:49.646600 6659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:13:49.649133 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:49.638169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34831 (local address 127.0.0.1:35007)
0807 03:13:49.638369 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:49.638383 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:49.642790 (+ 4407us) server_negotiation.cc:407] Connection header received
0807 03:13:49.642993 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:49.642998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:49.643080 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:49.643195 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:49.643697 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:49.644814 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:49.646436 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:49.646955 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:49.647109 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:49.648253 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:49.648262 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:49.648266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:49.648288 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:49.648382 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:49.648404 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:49.648540 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:49.648734 (+ 194us) server_negotiation.cc:299] Negotiation successful
0807 03:13:49.648785 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 03:13:49.650900 6669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [term 1 LEADER]: Becoming Leader. State: Replica: 18a1ca4a30954618bcd0b12a23cab3bc, State: Running, Role: LEADER
I20260807 03:13:49.651556 6669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a1ca4a30954618bcd0b12a23cab3bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35007 } }
I20260807 03:13:49.655884 6671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18a1ca4a30954618bcd0b12a23cab3bc. Latest consensus state: current_term: 1 leader_uuid: "18a1ca4a30954618bcd0b12a23cab3bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a1ca4a30954618bcd0b12a23cab3bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35007 } } }
I20260807 03:13:49.656366 6671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:49.657323 6677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:13:49.657804 6670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18a1ca4a30954618bcd0b12a23cab3bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a1ca4a30954618bcd0b12a23cab3bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35007 } } }
I20260807 03:13:49.658134 6670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:49.662671 6677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:13:49.672484 6677 catalog_manager.cc:1347] Generated new cluster ID: 0f0fa4e0dc9f4975948935327d411a68
I20260807 03:13:49.672585 6677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:13:49.685009 6677 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:13:49.686615 6677 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:13:49.696512 6677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18a1ca4a30954618bcd0b12a23cab3bc: Generated new TSK 0
I20260807 03:13:49.697224 6677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:13:49.709887 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 6591
W20260807 03:13:49.710997 6654 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37551
W20260807 03:13:49.711721 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 6591: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:13:49.711908 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 6591
W20260807 03:13:49.729758 6576 connection.cc:535] client connection to 127.0.0.1:35007 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35007): Connection reset by peer (error 104)
2026-08-07T03:13:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:13:49 dist-test-slave-tm1g krb5kdc[6690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:13:49 dist-test-slave-tm1g krb5kdc[6690](info): set up 2 sockets
Aug 07 03:13:49 dist-test-slave-tm1g krb5kdc[6690](info): commencing operation
krb5kdc: starting...
W20260807 03:13:50.840698 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:13:51 dist-test-slave-tm1g krb5kdc[6690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:13:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:13:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:13:51.099255 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39395
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:51.426048 6705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:51.426406 6705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:51.426586 6705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:51.433897 6705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:13:51.434024 6705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:51.434122 6705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:51.434228 6705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:13:51.434289 6705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:13:51.444067 6705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39395
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:51.445372 6705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:51.446738 6705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:51.458760 6710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:51.458828 6713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:51.458835 6711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:51.459589 6705 server_base.cc:1034] running on GCE node
I20260807 03:13:51.460244 6705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:51.461508 6705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:51.462705 6705 hybrid_clock.cc:648] HybridClock initialized: now 1786072431462658 us; error 36 us; skew 500 ppm
I20260807 03:13:51.463053 6705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:51 dist-test-slave-tm1g krb5kdc[6690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:51.471187 6705 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:13:51.472674 6705 webserver.cc:460] Webserver started at http://127.0.0.1:43969/ using document root <none> and password file <none>
I20260807 03:13:51.473341 6705 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:51.473495 6705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:51.473798 6705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:51.476794 6705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "a555eb97ccc442528d36a2e84af0cceb"
format_stamp: "Formatted at 2026-08-07 03:13:51 on dist-test-slave-tm1g"
server_key: "2bbdbc10b6bab5e85949be3cebd22f7d"
server_key_iv: "7b9c056b865b110b03e69c6902ed47ec"
server_key_version: "encryptionkey@0"
I20260807 03:13:51.477576 6705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "a555eb97ccc442528d36a2e84af0cceb"
format_stamp: "Formatted at 2026-08-07 03:13:51 on dist-test-slave-tm1g"
server_key: "2bbdbc10b6bab5e85949be3cebd22f7d"
server_key_iv: "7b9c056b865b110b03e69c6902ed47ec"
server_key_version: "encryptionkey@0"
I20260807 03:13:51.482741 6705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260807 03:13:51.485986 6720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:51.487355 6705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:13:51.487551 6705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "a555eb97ccc442528d36a2e84af0cceb"
format_stamp: "Formatted at 2026-08-07 03:13:51 on dist-test-slave-tm1g"
server_key: "2bbdbc10b6bab5e85949be3cebd22f7d"
server_key_iv: "7b9c056b865b110b03e69c6902ed47ec"
server_key_version: "encryptionkey@0"
I20260807 03:13:51.487752 6705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:51.493281 6705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:51.497511 6705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:51.497810 6705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:51.515748 6705 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39395
I20260807 03:13:51.515827 6771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39395 every 8 connection(s)
I20260807 03:13:51.517112 6705 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:13:51.520983 6772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:51.523675 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 6705
I20260807 03:13:51.523856 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:13:51.524322 17532 external_mini_cluster.cc:1428] Setting key 0197963a9c909fc273639416c1f80557
I20260807 03:13:51.533107 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:51.526106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59351 (local address 127.0.0.1:39395)
0807 03:13:51.526985 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:51.527000 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:51.527047 (+ 47us) server_negotiation.cc:407] Connection header received
0807 03:13:51.529334 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:51.529365 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:51.529805 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:51.530236 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:51.531242 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:51.531250 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:51.531267 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:51.531295 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:51.531363 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:51.531395 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:51.531620 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:51.531755 (+ 135us) server_negotiation.cc:299] Negotiation successful
0807 03:13:51.531950 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":384,"threads_started":1}
I20260807 03:13:51.533952 6772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb: Bootstrap starting.
I20260807 03:13:51.536819 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:13:51.537489 6772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:51.538529 6772 log.cc:824] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:51.541671 6772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb: No bootstrap required, opened a new log
I20260807 03:13:51.543766 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:51.541253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55309 (local address 127.0.0.1:39395)
0807 03:13:51.541397 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:51.541401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:51.541841 (+ 440us) server_negotiation.cc:407] Connection header received
0807 03:13:51.542017 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:51.542023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:51.542082 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:51.542187 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:51.543112 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:51.543117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:51.543125 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:51.543145 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:51.543189 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:51.543205 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:51.543340 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:51.543449 (+ 109us) server_negotiation.cc:299] Negotiation successful
0807 03:13:51.543509 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 03:13:51.549957 6772 raft_consensus.cc:348] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a555eb97ccc442528d36a2e84af0cceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } }
I20260807 03:13:51.550309 6772 raft_consensus.cc:374] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:51.550448 6772 raft_consensus.cc:729] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a555eb97ccc442528d36a2e84af0cceb, State: Initialized, Role: FOLLOWER
I20260807 03:13:51.551039 6772 consensus_queue.cc:260] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a555eb97ccc442528d36a2e84af0cceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } }
I20260807 03:13:51.551277 6772 raft_consensus.cc:388] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:13:51.551424 6772 raft_consensus.cc:482] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:13:51.551599 6772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:51.553751 6772 raft_consensus.cc:504] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a555eb97ccc442528d36a2e84af0cceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } }
I20260807 03:13:51.554221 6772 leader_election.cc:302] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a555eb97ccc442528d36a2e84af0cceb; no voters:
I20260807 03:13:51.554896 6772 leader_election.cc:288] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:13:51.555027 6784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:51.556085 6784 raft_consensus.cc:686] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [term 1 LEADER]: Becoming Leader. State: Replica: a555eb97ccc442528d36a2e84af0cceb, State: Running, Role: LEADER
I20260807 03:13:51.556559 6784 consensus_queue.cc:237] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a555eb97ccc442528d36a2e84af0cceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } }
I20260807 03:13:51.556963 6772 sys_catalog.cc:564] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:13:51.559286 6785 sys_catalog.cc:455] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a555eb97ccc442528d36a2e84af0cceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a555eb97ccc442528d36a2e84af0cceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } } }
I20260807 03:13:51.559751 6785 sys_catalog.cc:458] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:51.559578 6786 sys_catalog.cc:455] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a555eb97ccc442528d36a2e84af0cceb. Latest consensus state: current_term: 1 leader_uuid: "a555eb97ccc442528d36a2e84af0cceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a555eb97ccc442528d36a2e84af0cceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39395 } } }
I20260807 03:13:51.561146 6786 sys_catalog.cc:458] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:51.562659 6792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:13:51.567809 6792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:13:51.576336 6792 catalog_manager.cc:1347] Generated new cluster ID: 9705008f2abb4dce86fba7f1669b7be0
I20260807 03:13:51.576741 6792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:13:51.605406 6792 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:13:51.605824 6792 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:13:51.624185 6792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a555eb97ccc442528d36a2e84af0cceb: Generated new TSK 0
I20260807 03:13:51.624991 6792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:13:51.640997 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 6705
W20260807 03:13:51.642279 6767 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59351
W20260807 03:13:51.643020 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 6705: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:13:51.643167 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 6705
2026-08-07T03:13:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1930 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:13:51 dist-test-slave-tm1g krb5kdc[6806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:13:51 dist-test-slave-tm1g krb5kdc[6806](info): set up 2 sockets
Aug 07 03:13:51 dist-test-slave-tm1g krb5kdc[6806](info): commencing operation
krb5kdc: starting...
W20260807 03:13:52.780529 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:13:52 dist-test-slave-tm1g krb5kdc[6806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:13:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:13:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:13:53.055404 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:53.386588 6819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:53.386868 6819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:53.387061 6819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:53.394477 6819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:13:53.394603 6819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:53.394695 6819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:53.394779 6819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:13:53.394865 6819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:13:53.404495 6819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:53.405803 6819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:53.407160 6819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:53.419286 6824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:53.419332 6827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:53.419363 6825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:53.420528 6819 server_base.cc:1034] running on GCE node
I20260807 03:13:53.421257 6819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:53.422662 6819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:53.423887 6819 hybrid_clock.cc:648] HybridClock initialized: now 1786072433423856 us; error 42 us; skew 500 ppm
I20260807 03:13:53.424230 6819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:53 dist-test-slave-tm1g krb5kdc[6806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:53.427470 6819 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:13:53.429024 6819 webserver.cc:460] Webserver started at http://127.0.0.1:44365/ using document root <none> and password file <none>
I20260807 03:13:53.429708 6819 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:53.429857 6819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:53.430173 6819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:53.433120 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "398af92a623544e2a4254493dd26f987"
format_stamp: "Formatted at 2026-08-07 03:13:53 on dist-test-slave-tm1g"
server_key: "9d01a455780b9bf2bc7a1e86d3e6c522"
server_key_iv: "3611ec7b74e8d0e412a9c8480218df72"
server_key_version: "encryptionkey@0"
I20260807 03:13:53.433884 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "398af92a623544e2a4254493dd26f987"
format_stamp: "Formatted at 2026-08-07 03:13:53 on dist-test-slave-tm1g"
server_key: "9d01a455780b9bf2bc7a1e86d3e6c522"
server_key_iv: "3611ec7b74e8d0e412a9c8480218df72"
server_key_version: "encryptionkey@0"
I20260807 03:13:53.438979 6819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260807 03:13:53.442233 6834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:53.443567 6819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:13:53.443760 6819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "398af92a623544e2a4254493dd26f987"
format_stamp: "Formatted at 2026-08-07 03:13:53 on dist-test-slave-tm1g"
server_key: "9d01a455780b9bf2bc7a1e86d3e6c522"
server_key_iv: "3611ec7b74e8d0e412a9c8480218df72"
server_key_version: "encryptionkey@0"
I20260807 03:13:53.443959 6819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:53.477344 6819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:53.481793 6819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:53.482095 6819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:53.500290 6819 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32849
I20260807 03:13:53.500332 6887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32849 every 8 connection(s)
I20260807 03:13:53.501695 6819 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:13:53.505362 6888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:53.509927 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 6819
I20260807 03:13:53.510107 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:13:53.510619 17532 external_mini_cluster.cc:1428] Setting key b72b8e7f5221b1d8965034acf9ccef08
I20260807 03:13:53.517431 6888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987: Bootstrap starting.
I20260807 03:13:53.521054 6888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:53.522264 6888 log.cc:824] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:53.525104 6888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987: No bootstrap required, opened a new log
Aug 07 03:13:53 dist-test-slave-tm1g krb5kdc[6806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 03:13:53.535519 6891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:53.512406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44889 (local address 127.0.0.1:32849)
0807 03:13:53.513263 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:53.513282 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:53.513332 (+ 50us) server_negotiation.cc:407] Connection header received
0807 03:13:53.515869 (+ 2537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:53.515896 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:53.516351 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:53.516772 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:53.518101 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:53.520201 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:53.521520 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:53.522592 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:53.522785 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:53.527013 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:53.527025 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:53.527044 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:53.527102 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:53.531163 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:53.531825 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:53.531833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:53.531841 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:53.532054 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:53.532400 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:53.532409 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:53.532413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:53.533116 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:53.533417 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:53.533725 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 03:13:53.534186 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":351,"threads_started":1}
I20260807 03:13:53.535627 6888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "398af92a623544e2a4254493dd26f987" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } }
I20260807 03:13:53.536053 6888 raft_consensus.cc:374] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:53.536192 6888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398af92a623544e2a4254493dd26f987, State: Initialized, Role: FOLLOWER
I20260807 03:13:53.536767 6888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "398af92a623544e2a4254493dd26f987" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } }
I20260807 03:13:53.537016 6888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:13:53.537173 6888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:13:53.537392 6888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:53.539484 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:13:53.540040 6888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "398af92a623544e2a4254493dd26f987" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } }
I20260807 03:13:53.540558 6888 leader_election.cc:302] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 398af92a623544e2a4254493dd26f987; no voters:
I20260807 03:13:53.541179 6888 leader_election.cc:288] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:13:53.541290 6893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:53.542771 6893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [term 1 LEADER]: Becoming Leader. State: Replica: 398af92a623544e2a4254493dd26f987, State: Running, Role: LEADER
I20260807 03:13:53.543293 6893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "398af92a623544e2a4254493dd26f987" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } }
I20260807 03:13:53.543529 6888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:13:53.545957 6895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 398af92a623544e2a4254493dd26f987. Latest consensus state: current_term: 1 leader_uuid: "398af92a623544e2a4254493dd26f987" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "398af92a623544e2a4254493dd26f987" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } } }
I20260807 03:13:53.546393 6895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:53.546051 6894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "398af92a623544e2a4254493dd26f987" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "398af92a623544e2a4254493dd26f987" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32849 } } }
I20260807 03:13:53.547021 6894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987 [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:53.547399 6902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:13:53.564280 6902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:13:53.580581 6902 catalog_manager.cc:1347] Generated new cluster ID: 0de8d70b886243b8b7096e29626e08a5
I20260807 03:13:53.580722 6902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:13:53.583052 6891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:53.560278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55629 (local address 127.0.0.1:32849)
0807 03:13:53.560467 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:53.560475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:53.562931 (+ 2456us) server_negotiation.cc:407] Connection header received
0807 03:13:53.563255 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:53.563265 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:53.563370 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:53.563456 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:53.566424 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:53.567961 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:53.570062 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:53.571034 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:53.571165 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:53.575308 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:53.575318 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:53.575326 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:53.575376 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:53.579548 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:53.580443 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:53.580449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:53.580452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:53.580708 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:53.581207 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:53.581212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:53.581219 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:53.581666 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:53.581895 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:53.582498 (+ 603us) server_negotiation.cc:299] Negotiation successful
0807 03:13:53.582701 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 03:13:53.607136 6902 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:13:53.608407 6902 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:13:53.614555 6902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 398af92a623544e2a4254493dd26f987: Generated new TSK 0
I20260807 03:13:53.615307 6902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:13:53.623610 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 6819
I20260807 03:13:53.624735 6883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:53.715565 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 6819
2026-08-07T03:13:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2073 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:13:53 dist-test-slave-tm1g krb5kdc[6920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:13:53 dist-test-slave-tm1g krb5kdc[6920](info): set up 2 sockets
Aug 07 03:13:53 dist-test-slave-tm1g krb5kdc[6920](info): commencing operation
krb5kdc: starting...
W20260807 03:13:54.847869 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:13:55 dist-test-slave-tm1g krb5kdc[6920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:13:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:13:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:13:55.103875 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:55.424772 6933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:55.425047 6933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:55.425168 6933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:55.432178 6933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:13:55.432286 6933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:55.432351 6933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:55.432406 6933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:13:55.432504 6933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:13:55.441996 6933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:55.443259 6933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:55.444554 6933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:55.456498 6941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:55.456431 6939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:55.456854 6938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:55.457229 6933 server_base.cc:1034] running on GCE node
I20260807 03:13:55.457993 6933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:55.459453 6933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:55.460686 6933 hybrid_clock.cc:648] HybridClock initialized: now 1786072435460634 us; error 40 us; skew 500 ppm
I20260807 03:13:55.461045 6933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:55 dist-test-slave-tm1g krb5kdc[6920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:55.464198 6933 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:13:55.465579 6933 webserver.cc:460] Webserver started at http://127.0.0.1:45523/ using document root <none> and password file <none>
I20260807 03:13:55.466197 6933 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:55.466347 6933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:55.466621 6933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:55.469522 6933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "4669db75ab954594a3a7bf0bee11f26c"
format_stamp: "Formatted at 2026-08-07 03:13:55 on dist-test-slave-tm1g"
server_key: "1f6c7112a1e4d3a12af2ff00a7e13d4f"
server_key_iv: "db68d667894d3e65c90701fd9f0aef1e"
server_key_version: "encryptionkey@0"
I20260807 03:13:55.470276 6933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "4669db75ab954594a3a7bf0bee11f26c"
format_stamp: "Formatted at 2026-08-07 03:13:55 on dist-test-slave-tm1g"
server_key: "1f6c7112a1e4d3a12af2ff00a7e13d4f"
server_key_iv: "db68d667894d3e65c90701fd9f0aef1e"
server_key_version: "encryptionkey@0"
I20260807 03:13:55.475350 6933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260807 03:13:55.478623 6948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:55.479863 6933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:13:55.480000 6933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "4669db75ab954594a3a7bf0bee11f26c"
format_stamp: "Formatted at 2026-08-07 03:13:55 on dist-test-slave-tm1g"
server_key: "1f6c7112a1e4d3a12af2ff00a7e13d4f"
server_key_iv: "db68d667894d3e65c90701fd9f0aef1e"
server_key_version: "encryptionkey@0"
I20260807 03:13:55.480162 6933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:55.497558 6933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:55.501422 6933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:55.501662 6933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:55.520146 6933 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32893
I20260807 03:13:55.520183 7000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32893 every 8 connection(s)
I20260807 03:13:55.521490 6933 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:13:55.525079 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:55.528398 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 6933
I20260807 03:13:55.528596 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:13:55.529125 17532 external_mini_cluster.cc:1428] Setting key 35465b388bcef98b00d8d52a8dcb1765
I20260807 03:13:55.537005 7001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c: Bootstrap starting.
I20260807 03:13:55.540505 7001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:55.541680 7001 log.cc:824] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:55.542166 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:55.530934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35981 (local address 127.0.0.1:32893)
0807 03:13:55.531649 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:55.531659 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:55.531698 (+ 39us) server_negotiation.cc:407] Connection header received
0807 03:13:55.533282 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:55.533303 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:55.533707 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:55.534010 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:55.534625 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:55.535919 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:55.537799 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:55.538540 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:55.538682 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:55.539885 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:55.539892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:55.539903 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:55.540407 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:55.540475 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:55.540502 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:55.540645 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:55.540844 (+ 199us) server_negotiation.cc:299] Negotiation successful
0807 03:13:55.541115 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":329,"threads_started":1}
I20260807 03:13:55.544688 7001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c: No bootstrap required, opened a new log
I20260807 03:13:55.545948 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:13:55.552707 7001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669db75ab954594a3a7bf0bee11f26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32893 } }
I20260807 03:13:55.553097 7001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:55.553225 7001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4669db75ab954594a3a7bf0bee11f26c, State: Initialized, Role: FOLLOWER
I20260807 03:13:55.553746 7001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669db75ab954594a3a7bf0bee11f26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32893 } }
I20260807 03:13:55.553973 7001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:13:55.554113 7001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:13:55.554301 7001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:55.556766 7001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669db75ab954594a3a7bf0bee11f26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32893 } }
I20260807 03:13:55.556939 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:55.550358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57207 (local address 127.0.0.1:32893)
0807 03:13:55.550542 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:55.550547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:55.550961 (+ 414us) server_negotiation.cc:407] Connection header received
0807 03:13:55.551098 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:55.551104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:55.551181 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:55.551291 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:55.551735 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:55.552756 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:55.554553 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:55.554998 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:55.555168 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:55.556217 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:55.556221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:55.556224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:55.556240 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:55.556291 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:55.556306 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:55.556397 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:55.556521 (+ 124us) server_negotiation.cc:299] Negotiation successful
0807 03:13:55.556627 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 03:13:55.557313 7001 leader_election.cc:302] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4669db75ab954594a3a7bf0bee11f26c; no voters:
I20260807 03:13:55.557895 7001 leader_election.cc:288] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:13:55.558025 7014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:55.558826 7014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [term 1 LEADER]: Becoming Leader. State: Replica: 4669db75ab954594a3a7bf0bee11f26c, State: Running, Role: LEADER
I20260807 03:13:55.559450 7014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669db75ab954594a3a7bf0bee11f26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32893 } }
I20260807 03:13:55.560216 7001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:13:55.562215 7015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4669db75ab954594a3a7bf0bee11f26c. Latest consensus state: current_term: 1 leader_uuid: "4669db75ab954594a3a7bf0bee11f26c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669db75ab954594a3a7bf0bee11f26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32893 } } }
I20260807 03:13:55.562667 7015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:55.563640 7016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4669db75ab954594a3a7bf0bee11f26c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669db75ab954594a3a7bf0bee11f26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32893 } } }
I20260807 03:13:55.564052 7016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:55.568548 7021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:13:55.573340 7021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:13:55.581885 7021 catalog_manager.cc:1347] Generated new cluster ID: 026b06bd6e6547e295edd4f06ff82cee
I20260807 03:13:55.581986 7021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:13:55.605911 7021 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:13:55.607473 7021 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:13:55.622555 7021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4669db75ab954594a3a7bf0bee11f26c: Generated new TSK 0
I20260807 03:13:55.623557 7021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:13:55.630280 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 6933
W20260807 03:13:55.631374 6996 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35981
W20260807 03:13:55.632012 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 6933: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:13:55.632166 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 6933
2026-08-07T03:13:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1915 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:13:55 dist-test-slave-tm1g krb5kdc[7035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:13:55 dist-test-slave-tm1g krb5kdc[7035](info): set up 2 sockets
Aug 07 03:13:55 dist-test-slave-tm1g krb5kdc[7035](info): commencing operation
krb5kdc: starting...
W20260807 03:13:56.759752 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:13:56 dist-test-slave-tm1g krb5kdc[7035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:13:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:13:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:13:57.021588 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:57.348500 7049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:57.348766 7049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:57.348856 7049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:57.355672 7049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:13:57.355770 7049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:57.355816 7049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:57.355875 7049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:13:57.355917 7049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:13:57.365228 7049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:57.366525 7049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:57.367735 7049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:57.379187 7054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:57.379215 7057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:57.379060 7055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:57.380084 7049 server_base.cc:1034] running on GCE node
I20260807 03:13:57.380822 7049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:57.382227 7049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:57.383474 7049 hybrid_clock.cc:648] HybridClock initialized: now 1786072437383432 us; error 37 us; skew 500 ppm
I20260807 03:13:57.383813 7049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:57 dist-test-slave-tm1g krb5kdc[7035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:57.392782 7049 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:13:57.394825 7049 webserver.cc:460] Webserver started at http://127.0.0.1:36069/ using document root <none> and password file <none>
I20260807 03:13:57.395751 7049 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:57.395910 7049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:57.396297 7049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:57.400610 7049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "5c1f957771a946668290426d6c1f087f"
format_stamp: "Formatted at 2026-08-07 03:13:57 on dist-test-slave-tm1g"
server_key: "15f75237d460ecd34ac20c9b0b8deb8e"
server_key_iv: "21e9a0d99b24bc3397bd5316273f4253"
server_key_version: "encryptionkey@0"
I20260807 03:13:57.401752 7049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "5c1f957771a946668290426d6c1f087f"
format_stamp: "Formatted at 2026-08-07 03:13:57 on dist-test-slave-tm1g"
server_key: "15f75237d460ecd34ac20c9b0b8deb8e"
server_key_iv: "21e9a0d99b24bc3397bd5316273f4253"
server_key_version: "encryptionkey@0"
I20260807 03:13:57.409198 7049 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260807 03:13:57.414191 7064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:57.415963 7049 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 03:13:57.416172 7049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "5c1f957771a946668290426d6c1f087f"
format_stamp: "Formatted at 2026-08-07 03:13:57 on dist-test-slave-tm1g"
server_key: "15f75237d460ecd34ac20c9b0b8deb8e"
server_key_iv: "21e9a0d99b24bc3397bd5316273f4253"
server_key_version: "encryptionkey@0"
I20260807 03:13:57.416425 7049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:57.422814 7049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:57.427496 7049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:57.427819 7049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:57.448578 7049 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34057
I20260807 03:13:57.448619 7117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34057 every 8 connection(s)
I20260807 03:13:57.449939 7049 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:13:57.453527 7118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:57.455284 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 7049
I20260807 03:13:57.455476 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:13:57.456061 17532 external_mini_cluster.cc:1428] Setting key 3fdd781dfe4ac6f960e826b121a7c1a4
I20260807 03:13:57.463644 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:57.457808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55213 (local address 127.0.0.1:34057)
0807 03:13:57.458669 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:57.458684 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:57.458728 (+ 44us) server_negotiation.cc:407] Connection header received
0807 03:13:57.460275 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:57.460295 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:57.460622 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:57.460935 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:57.462010 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:57.462015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:57.462027 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:57.462043 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:57.462101 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:57.462124 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:57.462284 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:57.462547 (+ 263us) server_negotiation.cc:299] Negotiation successful
0807 03:13:57.462749 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":435,"threads_started":1}
I20260807 03:13:57.467053 7118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f: Bootstrap starting.
I20260807 03:13:57.467353 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:13:57.470793 7118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:57.472148 7118 log.cc:824] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:57.475411 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:57.472887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49545 (local address 127.0.0.1:34057)
0807 03:13:57.473119 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:57.473125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:57.473482 (+ 357us) server_negotiation.cc:407] Connection header received
0807 03:13:57.473669 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:57.473674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:57.473730 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:57.473839 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:57.474765 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:57.474773 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:57.474781 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:13:57.474804 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:57.474854 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:13:57.474877 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:57.474991 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:57.475089 (+ 98us) server_negotiation.cc:299] Negotiation successful
0807 03:13:57.475148 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 03:13:57.475720 7118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f: No bootstrap required, opened a new log
I20260807 03:13:57.482817 7118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1f957771a946668290426d6c1f087f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } }
I20260807 03:13:57.483187 7118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:57.483299 7118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c1f957771a946668290426d6c1f087f, State: Initialized, Role: FOLLOWER
I20260807 03:13:57.483827 7118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1f957771a946668290426d6c1f087f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } }
I20260807 03:13:57.484061 7118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:13:57.484207 7118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:13:57.484369 7118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:57.486661 7118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1f957771a946668290426d6c1f087f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } }
I20260807 03:13:57.487144 7118 leader_election.cc:302] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c1f957771a946668290426d6c1f087f; no voters:
I20260807 03:13:57.487666 7118 leader_election.cc:288] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:13:57.487769 7128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:57.489233 7128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [term 1 LEADER]: Becoming Leader. State: Replica: 5c1f957771a946668290426d6c1f087f, State: Running, Role: LEADER
I20260807 03:13:57.489773 7118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:13:57.489705 7128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1f957771a946668290426d6c1f087f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } }
I20260807 03:13:57.492520 7130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c1f957771a946668290426d6c1f087f. Latest consensus state: current_term: 1 leader_uuid: "5c1f957771a946668290426d6c1f087f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1f957771a946668290426d6c1f087f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } } }
I20260807 03:13:57.492940 7130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:57.493525 7129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c1f957771a946668290426d6c1f087f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c1f957771a946668290426d6c1f087f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34057 } } }
I20260807 03:13:57.494065 7137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:13:57.494067 7129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:57.501636 7137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:13:57.513234 7137 catalog_manager.cc:1347] Generated new cluster ID: ada40e49692245b9bd7b8f7fdc6b0451
I20260807 03:13:57.513338 7137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:13:57.540305 7137 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:13:57.540731 7137 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:13:57.549950 7137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c1f957771a946668290426d6c1f087f: Generated new TSK 0
I20260807 03:13:57.550769 7137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:13:57.556408 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 7049
W20260807 03:13:57.557510 7113 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55213
W20260807 03:13:57.558189 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 7049: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:13:57.558370 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 7049
2026-08-07T03:13:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:13:57 dist-test-slave-tm1g krb5kdc[7150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:13:57 dist-test-slave-tm1g krb5kdc[7150](info): set up 2 sockets
Aug 07 03:13:57 dist-test-slave-tm1g krb5kdc[7150](info): commencing operation
krb5kdc: starting...
W20260807 03:13:58.688169 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:13:58 dist-test-slave-tm1g krb5kdc[7150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:13:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:13:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:13:58.958693 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43445
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:13:59.281929 7163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:13:59.282184 7163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:13:59.282275 7163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:13:59.289345 7163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:13:59.289456 7163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:13:59.289501 7163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:13:59.289553 7163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:13:59.289588 7163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:13:59.298827 7163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:13:59.299973 7163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:13:59.301203 7163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:13:59.312662 7168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:59.312722 7171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:13:59.312687 7169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:13:59.313660 7163 server_base.cc:1034] running on GCE node
I20260807 03:13:59.314432 7163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:13:59.315714 7163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:13:59.316921 7163 hybrid_clock.cc:648] HybridClock initialized: now 1786072439316884 us; error 33 us; skew 500 ppm
I20260807 03:13:59.317276 7163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:13:59 dist-test-slave-tm1g krb5kdc[7150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:13:59.320323 7163 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:13:59.321732 7163 webserver.cc:460] Webserver started at http://127.0.0.1:36785/ using document root <none> and password file <none>
I20260807 03:13:59.322440 7163 fs_manager.cc:359] Metadata directory not provided
I20260807 03:13:59.322554 7163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:13:59.322808 7163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:13:59.325644 7163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "6d36649cb5a540238fdd30ea97945190"
format_stamp: "Formatted at 2026-08-07 03:13:59 on dist-test-slave-tm1g"
server_key: "b5db16bd09ba4b67e9d28a6678418c22"
server_key_iv: "16f6e2db5572e607bf264bae6cf4c2eb"
server_key_version: "encryptionkey@0"
I20260807 03:13:59.326416 7163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "6d36649cb5a540238fdd30ea97945190"
format_stamp: "Formatted at 2026-08-07 03:13:59 on dist-test-slave-tm1g"
server_key: "b5db16bd09ba4b67e9d28a6678418c22"
server_key_iv: "16f6e2db5572e607bf264bae6cf4c2eb"
server_key_version: "encryptionkey@0"
I20260807 03:13:59.331269 7163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 03:13:59.334570 7178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:13:59.335811 7163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:13:59.335990 7163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "6d36649cb5a540238fdd30ea97945190"
format_stamp: "Formatted at 2026-08-07 03:13:59 on dist-test-slave-tm1g"
server_key: "b5db16bd09ba4b67e9d28a6678418c22"
server_key_iv: "16f6e2db5572e607bf264bae6cf4c2eb"
server_key_version: "encryptionkey@0"
I20260807 03:13:59.336189 7163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:13:59.348801 7163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:13:59.352520 7163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:13:59.352803 7163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:13:59.371053 7163 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43445
I20260807 03:13:59.371089 7230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43445 every 8 connection(s)
I20260807 03:13:59.372468 7163 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:13:59.376008 7231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:13:59.382151 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 7163
I20260807 03:13:59.382375 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:13:59.382822 17532 external_mini_cluster.cc:1428] Setting key 9ff13c972390614dc3f8a04c526ba608
I20260807 03:13:59.386289 7231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190: Bootstrap starting.
I20260807 03:13:59.389971 7231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190: Neither blocks nor log segments found. Creating new log.
I20260807 03:13:59.391073 7231 log.cc:824] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190: Log is configured to *not* fsync() on all Append() calls
I20260807 03:13:59.394074 7231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190: No bootstrap required, opened a new log
Aug 07 03:13:59 dist-test-slave-tm1g krb5kdc[7150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 03:13:59.404749 7231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d36649cb5a540238fdd30ea97945190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43445 } }
I20260807 03:13:59.405267 7231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:13:59.405411 7231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d36649cb5a540238fdd30ea97945190, State: Initialized, Role: FOLLOWER
I20260807 03:13:59.406090 7231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d36649cb5a540238fdd30ea97945190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43445 } }
I20260807 03:13:59.406447 7231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:13:59.406592 7231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:13:59.406769 7231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:13:59.407202 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:59.384602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:55489 (local address 127.0.0.1:43445)
0807 03:13:59.385347 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:59.385357 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:59.385398 (+ 41us) server_negotiation.cc:407] Connection header received
0807 03:13:59.387125 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:59.387149 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:59.387513 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:59.387850 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:59.389692 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:59.391124 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:59.392792 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:59.393433 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:59.393578 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:59.398533 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:59.398544 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:59.398563 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:59.398612 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:59.402560 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:59.403290 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:59.403299 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:59.403308 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:59.403534 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:59.403967 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:59.403973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:59.403976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:59.404677 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:59.404990 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:59.405420 (+ 430us) server_negotiation.cc:299] Negotiation successful
0807 03:13:59.405762 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":352,"threads_started":1}
I20260807 03:13:59.409235 7231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d36649cb5a540238fdd30ea97945190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43445 } }
I20260807 03:13:59.409788 7231 leader_election.cc:302] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d36649cb5a540238fdd30ea97945190; no voters:
I20260807 03:13:59.410488 7231 leader_election.cc:288] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:13:59.410604 7236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:13:59.411168 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:13:59.412688 7231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:13:59.414609 7236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [term 1 LEADER]: Becoming Leader. State: Replica: 6d36649cb5a540238fdd30ea97945190, State: Running, Role: LEADER
I20260807 03:13:59.415225 7236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d36649cb5a540238fdd30ea97945190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43445 } }
I20260807 03:13:59.417976 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d36649cb5a540238fdd30ea97945190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d36649cb5a540238fdd30ea97945190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43445 } } }
I20260807 03:13:59.418354 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:59.417698 7238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d36649cb5a540238fdd30ea97945190. Latest consensus state: current_term: 1 leader_uuid: "6d36649cb5a540238fdd30ea97945190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d36649cb5a540238fdd30ea97945190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43445 } } }
I20260807 03:13:59.418954 7238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190 [sys.catalog]: This master's current role is: LEADER
I20260807 03:13:59.422669 7244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:13:59.428614 7244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:13:59.441811 7244 catalog_manager.cc:1347] Generated new cluster ID: 1cf9ed4a1363461cb4a81713a64e4b41
I20260807 03:13:59.441954 7244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:13:59.454563 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:13:59.426437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:34247 (local address 127.0.0.1:43445)
0807 03:13:59.426770 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0807 03:13:59.426779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:13:59.437129 (+ 10350us) server_negotiation.cc:407] Connection header received
0807 03:13:59.437694 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:13:59.437702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:13:59.437810 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:13:59.437900 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:13:59.440454 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:59.441645 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:59.443552 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:13:59.444186 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:13:59.444266 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:13:59.447456 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:13:59.447463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:13:59.447469 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:13:59.447506 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:13:59.451263 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:59.452166 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:59.452172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:59.452175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:59.452387 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:13:59.452903 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:13:59.452909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:13:59.452913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:13:59.453284 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:13:59.453562 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0807 03:13:59.453939 (+ 377us) server_negotiation.cc:299] Negotiation successful
0807 03:13:59.454135 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260807 03:13:59.461519 7244 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:13:59.463392 7244 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:13:59.482169 7244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d36649cb5a540238fdd30ea97945190: Generated new TSK 0
I20260807 03:13:59.483132 7244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:13:59.494565 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 7163
I20260807 03:13:59.496169 7226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:13:59.586656 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 7163
W20260807 03:13:59.604749 7148 connection.cc:535] client connection to 127.0.0.1:43445 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43445): Connection reset by peer (error 104)
2026-08-07T03:13:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2028 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:13:59 dist-test-slave-tm1g krb5kdc[7264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:13:59 dist-test-slave-tm1g krb5kdc[7264](info): set up 2 sockets
Aug 07 03:13:59 dist-test-slave-tm1g krb5kdc[7264](info): commencing operation
krb5kdc: starting...
W20260807 03:14:00.728190 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:14:00 dist-test-slave-tm1g krb5kdc[7264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:14:00.995878 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:14:01.326687 7278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:01.326969 7278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:01.327073 7278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:14:01.334177 7278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:01.334285 7278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:14:01.334375 7278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:01.334458 7278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:01.334506 7278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:01.344116 7278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:01.345391 7278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:01.346720 7278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:14:01.358835 7284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:01.358892 7283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:01.358915 7286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:01.360101 7278 server_base.cc:1034] running on GCE node
I20260807 03:14:01.360838 7278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:01.362155 7278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:01.363379 7278 hybrid_clock.cc:648] HybridClock initialized: now 1786072441363340 us; error 61 us; skew 500 ppm
I20260807 03:14:01.363713 7278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:01 dist-test-slave-tm1g krb5kdc[7264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:01.366881 7278 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:14:01.368296 7278 webserver.cc:460] Webserver started at http://127.0.0.1:44019/ using document root <none> and password file <none>
I20260807 03:14:01.368903 7278 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:01.369006 7278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:01.369267 7278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:01.372174 7278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "7220bedb27004be2a07ce0e793af6b39"
format_stamp: "Formatted at 2026-08-07 03:14:01 on dist-test-slave-tm1g"
server_key: "6a090f2a50d36e85d03a271eac4d243e"
server_key_iv: "72e04ae072b988b8a420e1f28b7371c3"
server_key_version: "encryptionkey@0"
I20260807 03:14:01.372870 7278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "7220bedb27004be2a07ce0e793af6b39"
format_stamp: "Formatted at 2026-08-07 03:14:01 on dist-test-slave-tm1g"
server_key: "6a090f2a50d36e85d03a271eac4d243e"
server_key_iv: "72e04ae072b988b8a420e1f28b7371c3"
server_key_version: "encryptionkey@0"
I20260807 03:14:01.377919 7278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260807 03:14:01.381295 7293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:01.382612 7278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:14:01.382792 7278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "7220bedb27004be2a07ce0e793af6b39"
format_stamp: "Formatted at 2026-08-07 03:14:01 on dist-test-slave-tm1g"
server_key: "6a090f2a50d36e85d03a271eac4d243e"
server_key_iv: "72e04ae072b988b8a420e1f28b7371c3"
server_key_version: "encryptionkey@0"
I20260807 03:14:01.382982 7278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:01.401727 7278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:01.405553 7278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:01.405839 7278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:01.424559 7278 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42183
I20260807 03:14:01.424582 7346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42183 every 8 connection(s)
I20260807 03:14:01.425946 7278 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:14:01.429729 7347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:01.429915 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 7278
I20260807 03:14:01.430112 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:14:01.430706 17532 external_mini_cluster.cc:1428] Setting key 402325007af944affa100d3486670e14
I20260807 03:14:01.443781 7347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39: Bootstrap starting.
I20260807 03:14:01.444016 7350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:01.432460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:51413 (local address 127.0.0.1:42183)
0807 03:14:01.433353 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:01.433364 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:01.433404 (+ 40us) server_negotiation.cc:407] Connection header received
0807 03:14:01.435028 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:01.435052 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:01.435424 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:01.435761 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:01.436370 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:01.437675 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:01.439561 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:01.440221 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:01.440354 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:01.441686 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:01.441694 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:01.441706 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:14:01.442242 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:01.442372 (+ 130us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:14:01.442416 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:01.442549 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:01.442827 (+ 278us) server_negotiation.cc:299] Negotiation successful
0807 03:14:01.443025 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":472,"threads_started":1}
I20260807 03:14:01.447052 7347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:01.447793 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:14:01.448222 7347 log.cc:824] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39: Log is configured to *not* fsync() on all Append() calls
I20260807 03:14:01.451407 7347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39: No bootstrap required, opened a new log
I20260807 03:14:01.459611 7350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:01.452924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:36309 (local address 127.0.0.1:42183)
0807 03:14:01.453065 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:01.453069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:01.453466 (+ 397us) server_negotiation.cc:407] Connection header received
0807 03:14:01.453638 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:01.453642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:01.453697 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:01.453791 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:01.454413 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:01.455185 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:01.457001 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:01.457424 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:01.457512 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:01.458836 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:01.458841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:01.458844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:14:01.458861 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:01.458894 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:14:01.458908 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:01.458987 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:01.459212 (+ 225us) server_negotiation.cc:299] Negotiation successful
0807 03:14:01.459251 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260807 03:14:01.461138 7347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7220bedb27004be2a07ce0e793af6b39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } }
I20260807 03:14:01.461526 7347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:14:01.461640 7347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7220bedb27004be2a07ce0e793af6b39, State: Initialized, Role: FOLLOWER
I20260807 03:14:01.462249 7347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7220bedb27004be2a07ce0e793af6b39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } }
I20260807 03:14:01.462558 7347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:14:01.462710 7347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:14:01.462898 7347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:01.465090 7347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7220bedb27004be2a07ce0e793af6b39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } }
I20260807 03:14:01.465590 7347 leader_election.cc:302] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7220bedb27004be2a07ce0e793af6b39; no voters:
I20260807 03:14:01.466161 7347 leader_election.cc:288] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:14:01.466288 7357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:14:01.466977 7357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [term 1 LEADER]: Becoming Leader. State: Replica: 7220bedb27004be2a07ce0e793af6b39, State: Running, Role: LEADER
I20260807 03:14:01.467638 7357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7220bedb27004be2a07ce0e793af6b39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } }
I20260807 03:14:01.469619 7347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:14:01.470574 7358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7220bedb27004be2a07ce0e793af6b39. Latest consensus state: current_term: 1 leader_uuid: "7220bedb27004be2a07ce0e793af6b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7220bedb27004be2a07ce0e793af6b39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } } }
I20260807 03:14:01.470541 7359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7220bedb27004be2a07ce0e793af6b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7220bedb27004be2a07ce0e793af6b39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42183 } } }
I20260807 03:14:01.471011 7359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:01.474128 7358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:01.475226 7362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:14:01.480253 7362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:14:01.489670 7362 catalog_manager.cc:1347] Generated new cluster ID: bf18a29570fa4e98be0ab8852bbf6bb1
I20260807 03:14:01.489825 7362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:14:01.514586 7362 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:14:01.515862 7362 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:14:01.522672 7362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7220bedb27004be2a07ce0e793af6b39: Generated new TSK 0
I20260807 03:14:01.523422 7362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:14:01.535835 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 7278
W20260807 03:14:01.536968 7342 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.78:51413
W20260807 03:14:01.537716 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 7278: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:14:01.537868 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 7278
W20260807 03:14:01.556092 7262 connection.cc:535] client connection to 127.0.0.1:42183 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42183): Connection reset by peer (error 104)
2026-08-07T03:14:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:14:01 dist-test-slave-tm1g krb5kdc[7378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:14:01 dist-test-slave-tm1g krb5kdc[7378](info): set up 2 sockets
Aug 07 03:14:01 dist-test-slave-tm1g krb5kdc[7378](info): commencing operation
krb5kdc: starting...
W20260807 03:14:02.660631 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:14:02 dist-test-slave-tm1g krb5kdc[7378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:14:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:14:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:14:02.939132 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:14:03.263945 7391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:03.264262 7391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:03.264464 7391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:14:03.271359 7391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:03.271498 7391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:14:03.271605 7391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:03.271713 7391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:03.271797 7391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:03.281270 7391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:03.282588 7391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:03.283898 7391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:14:03.295495 7396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:03.295454 7397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:03.295562 7399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:03.296201 7391 server_base.cc:1034] running on GCE node
I20260807 03:14:03.296917 7391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:03.298220 7391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:03.299468 7391 hybrid_clock.cc:648] HybridClock initialized: now 1786072443299428 us; error 32 us; skew 500 ppm
I20260807 03:14:03.299809 7391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:03 dist-test-slave-tm1g krb5kdc[7378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:03.307643 7391 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:14:03.309059 7391 webserver.cc:460] Webserver started at http://127.0.0.1:37227/ using document root <none> and password file <none>
I20260807 03:14:03.309734 7391 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:03.309886 7391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:03.310202 7391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:03.313282 7391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "39524ad51489429482bb4a765b63b0c2"
format_stamp: "Formatted at 2026-08-07 03:14:03 on dist-test-slave-tm1g"
server_key: "62fcad5ac9b149d7ab18a784cdec5a76"
server_key_iv: "62af50525646c44f097ec13065f841c3"
server_key_version: "encryptionkey@0"
I20260807 03:14:03.314077 7391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "39524ad51489429482bb4a765b63b0c2"
format_stamp: "Formatted at 2026-08-07 03:14:03 on dist-test-slave-tm1g"
server_key: "62fcad5ac9b149d7ab18a784cdec5a76"
server_key_iv: "62af50525646c44f097ec13065f841c3"
server_key_version: "encryptionkey@0"
I20260807 03:14:03.319205 7391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260807 03:14:03.322564 7406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:03.323895 7391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:14:03.324075 7391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "39524ad51489429482bb4a765b63b0c2"
format_stamp: "Formatted at 2026-08-07 03:14:03 on dist-test-slave-tm1g"
server_key: "62fcad5ac9b149d7ab18a784cdec5a76"
server_key_iv: "62af50525646c44f097ec13065f841c3"
server_key_version: "encryptionkey@0"
I20260807 03:14:03.324266 7391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:03.329854 7391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:03.334120 7391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:03.334470 7391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:03.353233 7391 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40281
I20260807 03:14:03.353279 7459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40281 every 8 connection(s)
I20260807 03:14:03.354632 7391 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:14:03.358280 7460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:03.363793 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 7391
I20260807 03:14:03.363983 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:14:03.364423 17532 external_mini_cluster.cc:1428] Setting key 48d68770e39b63fd81328daee7c6705c
I20260807 03:14:03.369845 7460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2: Bootstrap starting.
I20260807 03:14:03.373335 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:03.366283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:47939 (local address 127.0.0.1:40281)
0807 03:14:03.367238 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:03.367252 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:03.367306 (+ 54us) server_negotiation.cc:407] Connection header received
0807 03:14:03.369613 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:03.369639 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:03.370044 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:03.370486 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:03.371437 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:03.371445 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:03.371467 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:14:03.371489 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:03.371568 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:14:03.371612 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:03.371830 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:03.371982 (+ 152us) server_negotiation.cc:299] Negotiation successful
0807 03:14:03.372178 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":375,"threads_started":1}
I20260807 03:14:03.373769 7460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:03.374903 7460 log.cc:824] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2: Log is configured to *not* fsync() on all Append() calls
I20260807 03:14:03.377099 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:14:03.378064 7460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2: No bootstrap required, opened a new log
I20260807 03:14:03.384760 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:03.382411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:37461 (local address 127.0.0.1:40281)
0807 03:14:03.382584 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:03.382591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:03.382979 (+ 388us) server_negotiation.cc:407] Connection header received
0807 03:14:03.383094 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:03.383100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:03.383168 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:03.383275 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:03.384147 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:03.384153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:03.384158 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:14:03.384177 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:03.384225 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:14:03.384244 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:03.384370 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:03.384432 (+ 62us) server_negotiation.cc:299] Negotiation successful
0807 03:14:03.384495 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:14:03.387817 7460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39524ad51489429482bb4a765b63b0c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260807 03:14:03.388232 7460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:14:03.388342 7460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39524ad51489429482bb4a765b63b0c2, State: Initialized, Role: FOLLOWER
I20260807 03:14:03.388888 7460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39524ad51489429482bb4a765b63b0c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260807 03:14:03.389114 7460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:14:03.389264 7460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:14:03.389439 7460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:03.391605 7460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39524ad51489429482bb4a765b63b0c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260807 03:14:03.392076 7460 leader_election.cc:302] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39524ad51489429482bb4a765b63b0c2; no voters:
I20260807 03:14:03.392702 7460 leader_election.cc:288] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:14:03.392835 7470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:14:03.393492 7470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 39524ad51489429482bb4a765b63b0c2, State: Running, Role: LEADER
I20260807 03:14:03.394078 7470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39524ad51489429482bb4a765b63b0c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260807 03:14:03.394734 7460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:14:03.397025 7472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39524ad51489429482bb4a765b63b0c2. Latest consensus state: current_term: 1 leader_uuid: "39524ad51489429482bb4a765b63b0c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39524ad51489429482bb4a765b63b0c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } } }
I20260807 03:14:03.405376 7472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:03.398442 7471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39524ad51489429482bb4a765b63b0c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39524ad51489429482bb4a765b63b0c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } } }
I20260807 03:14:03.405906 7471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:03.407419 7482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:14:03.411720 7482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:14:03.419858 7482 catalog_manager.cc:1347] Generated new cluster ID: aa58fc40c5444481a24ccea625ce7e0f
I20260807 03:14:03.419967 7482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:14:03.437999 7482 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:14:03.438481 7482 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:14:03.449347 7482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39524ad51489429482bb4a765b63b0c2: Generated new TSK 0
I20260807 03:14:03.450096 7482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:14:03.461378 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 7391
W20260807 03:14:03.462509 7455 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.78:47939
W20260807 03:14:03.463193 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 7391: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:14:03.463351 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 7391
2026-08-07T03:14:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1924 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:14:03 dist-test-slave-tm1g krb5kdc[7492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:14:03 dist-test-slave-tm1g krb5kdc[7492](info): set up 2 sockets
Aug 07 03:14:03 dist-test-slave-tm1g krb5kdc[7492](info): commencing operation
krb5kdc: starting...
W20260807 03:14:04.596127 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:14:04 dist-test-slave-tm1g krb5kdc[7492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:14:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:14:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:14:04.859200 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:14:05.183732 7505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:05.184027 7505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:05.184214 7505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:14:05.191293 7505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:05.191428 7505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:14:05.191529 7505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:05.191640 7505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:05.191717 7505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:05.201661 7505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43329
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:05.203085 7505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:05.204702 7505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:14:05.216410 7510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:05.216532 7511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:05.216565 7513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:05.217262 7505 server_base.cc:1034] running on GCE node
I20260807 03:14:05.218043 7505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:05.219455 7505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:05.220695 7505 hybrid_clock.cc:648] HybridClock initialized: now 1786072445220640 us; error 39 us; skew 500 ppm
I20260807 03:14:05.221038 7505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:05 dist-test-slave-tm1g krb5kdc[7492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:05.224182 7505 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:14:05.225498 7505 webserver.cc:460] Webserver started at http://127.0.0.1:36283/ using document root <none> and password file <none>
I20260807 03:14:05.226105 7505 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:05.226202 7505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:05.226514 7505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:05.229352 7505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "b935131b93d642eb8268325631e03cbc"
format_stamp: "Formatted at 2026-08-07 03:14:05 on dist-test-slave-tm1g"
server_key: "6574739dedf4d0161d858f8543c2586d"
server_key_iv: "fb84eb6aebdbe3a5d97ffeb89941364d"
server_key_version: "encryptionkey@0"
I20260807 03:14:05.230062 7505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "b935131b93d642eb8268325631e03cbc"
format_stamp: "Formatted at 2026-08-07 03:14:05 on dist-test-slave-tm1g"
server_key: "6574739dedf4d0161d858f8543c2586d"
server_key_iv: "fb84eb6aebdbe3a5d97ffeb89941364d"
server_key_version: "encryptionkey@0"
I20260807 03:14:05.235003 7505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260807 03:14:05.238371 7520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:05.239620 7505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 03:14:05.239758 7505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "b935131b93d642eb8268325631e03cbc"
format_stamp: "Formatted at 2026-08-07 03:14:05 on dist-test-slave-tm1g"
server_key: "6574739dedf4d0161d858f8543c2586d"
server_key_iv: "fb84eb6aebdbe3a5d97ffeb89941364d"
server_key_version: "encryptionkey@0"
I20260807 03:14:05.239915 7505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:05.262049 7505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:05.265805 7505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:05.266038 7505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:05.284178 7505 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43329
I20260807 03:14:05.284276 7572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43329 every 8 connection(s)
I20260807 03:14:05.285621 7505 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:14:05.289179 7573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:05.292881 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 7505
I20260807 03:14:05.293061 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:14:05.293617 17532 external_mini_cluster.cc:1428] Setting key 4f5e59b7c7defa3c37afa5af69e87247
I20260807 03:14:05.302090 7573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc: Bootstrap starting.
I20260807 03:14:05.305828 7573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:05.307202 7573 log.cc:824] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc: Log is configured to *not* fsync() on all Append() calls
Aug 07 03:14:05 dist-test-slave-tm1g krb5kdc[7492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 03:14:05.310046 7573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc: No bootstrap required, opened a new log
I20260807 03:14:05.317066 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:05.295554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:35893 (local address 127.0.0.1:43329)
0807 03:14:05.296301 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:05.296312 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:05.296351 (+ 39us) server_negotiation.cc:407] Connection header received
0807 03:14:05.298021 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:05.298043 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:05.298502 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:05.298831 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:05.300658 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:05.301977 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:05.303761 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:05.304425 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:05.304557 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:05.309668 (+ 5111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:05.309676 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:05.309688 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:05.309720 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:05.312871 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:05.313681 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:05.313687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:05.313692 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:05.313873 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:05.314405 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:05.314409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:05.314411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:05.314894 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:05.315261 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:05.315758 (+ 497us) server_negotiation.cc:299] Negotiation successful
0807 03:14:05.316091 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":354,"threads_started":1}
I20260807 03:14:05.317977 7573 raft_consensus.cc:348] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b935131b93d642eb8268325631e03cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43329 } }
I20260807 03:14:05.318342 7573 raft_consensus.cc:374] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:14:05.318496 7573 raft_consensus.cc:729] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b935131b93d642eb8268325631e03cbc, State: Initialized, Role: FOLLOWER
I20260807 03:14:05.319047 7573 consensus_queue.cc:260] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b935131b93d642eb8268325631e03cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43329 } }
I20260807 03:14:05.319300 7573 raft_consensus.cc:388] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:14:05.319469 7573 raft_consensus.cc:482] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:14:05.319657 7573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:05.321033 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:14:05.322394 7573 raft_consensus.cc:504] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b935131b93d642eb8268325631e03cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43329 } }
I20260807 03:14:05.322933 7573 leader_election.cc:302] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b935131b93d642eb8268325631e03cbc; no voters:
I20260807 03:14:05.323555 7573 leader_election.cc:288] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:14:05.323626 7578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:14:05.326012 7573 sys_catalog.cc:564] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:14:05.332291 7578 raft_consensus.cc:686] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [term 1 LEADER]: Becoming Leader. State: Replica: b935131b93d642eb8268325631e03cbc, State: Running, Role: LEADER
I20260807 03:14:05.333014 7578 consensus_queue.cc:237] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b935131b93d642eb8268325631e03cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43329 } }
I20260807 03:14:05.336385 7580 sys_catalog.cc:455] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b935131b93d642eb8268325631e03cbc. Latest consensus state: current_term: 1 leader_uuid: "b935131b93d642eb8268325631e03cbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b935131b93d642eb8268325631e03cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43329 } } }
I20260807 03:14:05.336768 7580 sys_catalog.cc:458] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:05.337342 7579 sys_catalog.cc:455] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b935131b93d642eb8268325631e03cbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b935131b93d642eb8268325631e03cbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43329 } } }
I20260807 03:14:05.337697 7579 sys_catalog.cc:458] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:05.340821 7594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:14:05.346427 7594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:14:05.358558 7594 catalog_manager.cc:1347] Generated new cluster ID: fd7efbb02e7149048f6762de2c3cf6e7
I20260807 03:14:05.358661 7594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:14:05.360111 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:05.341366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:49021 (local address 127.0.0.1:43329)
0807 03:14:05.341547 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:05.341553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:05.342783 (+ 1230us) server_negotiation.cc:407] Connection header received
0807 03:14:05.342972 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:05.342982 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:05.343095 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:05.343228 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:05.345527 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:05.346769 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:05.348714 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:05.349413 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:05.349522 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:05.352962 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:05.352969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:05.352973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:05.353009 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:05.356495 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:05.357390 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:05.357395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:05.357398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:05.357571 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:05.358163 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:05.358168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:05.358171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:05.358609 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:05.358867 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:05.359576 (+ 709us) server_negotiation.cc:299] Negotiation successful
0807 03:14:05.359768 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260807 03:14:05.383388 7594 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:14:05.384682 7594 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:14:05.396077 7594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b935131b93d642eb8268325631e03cbc: Generated new TSK 0
I20260807 03:14:05.397128 7594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:14:05.410063 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 7505
I20260807 03:14:05.411252 7568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:14:05.505057 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 7505
W20260807 03:14:05.523135 7490 connection.cc:535] client connection to 127.0.0.1:43329 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43329): Connection reset by peer (error 104)
2026-08-07T03:14:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2046 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:14:05 dist-test-slave-tm1g krb5kdc[7606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:14:05 dist-test-slave-tm1g krb5kdc[7606](info): set up 2 sockets
Aug 07 03:14:05 dist-test-slave-tm1g krb5kdc[7606](info): commencing operation
krb5kdc: starting...
W20260807 03:14:06.632172 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:14:06 dist-test-slave-tm1g krb5kdc[7606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:14:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:14:06.897214 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:14:07.227530 7619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:07.227833 7619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:07.227973 7619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:14:07.235198 7619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:07.235296 7619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:14:07.235433 7619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:07.235538 7619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:07.235613 7619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:07.245518 7619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44831
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:07.246827 7619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:07.248119 7619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:14:07.259886 7625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:07.259838 7628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:07.259927 7626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:07.260736 7619 server_base.cc:1034] running on GCE node
I20260807 03:14:07.261462 7619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:07.262835 7619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:07.264074 7619 hybrid_clock.cc:648] HybridClock initialized: now 1786072447264034 us; error 43 us; skew 500 ppm
I20260807 03:14:07.264417 7619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:07 dist-test-slave-tm1g krb5kdc[7606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:07.267438 7619 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:14:07.268800 7619 webserver.cc:460] Webserver started at http://127.0.0.1:37883/ using document root <none> and password file <none>
I20260807 03:14:07.269472 7619 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:07.269623 7619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:07.269922 7619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:07.272953 7619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "8953b28228f149ceb07df49e6e36843b"
format_stamp: "Formatted at 2026-08-07 03:14:07 on dist-test-slave-tm1g"
server_key: "568f78df00a6338a57176f061a0cad9f"
server_key_iv: "9d089494ad3ff648ee3cebe1cee638dc"
server_key_version: "encryptionkey@0"
I20260807 03:14:07.273722 7619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "8953b28228f149ceb07df49e6e36843b"
format_stamp: "Formatted at 2026-08-07 03:14:07 on dist-test-slave-tm1g"
server_key: "568f78df00a6338a57176f061a0cad9f"
server_key_iv: "9d089494ad3ff648ee3cebe1cee638dc"
server_key_version: "encryptionkey@0"
I20260807 03:14:07.278669 7619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260807 03:14:07.281912 7635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:07.283227 7619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:14:07.283414 7619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "8953b28228f149ceb07df49e6e36843b"
format_stamp: "Formatted at 2026-08-07 03:14:07 on dist-test-slave-tm1g"
server_key: "568f78df00a6338a57176f061a0cad9f"
server_key_iv: "9d089494ad3ff648ee3cebe1cee638dc"
server_key_version: "encryptionkey@0"
I20260807 03:14:07.283612 7619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:07.305116 7619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:07.308923 7619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:07.309211 7619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:07.327245 7619 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44831
I20260807 03:14:07.327277 7687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44831 every 8 connection(s)
I20260807 03:14:07.328559 7619 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:14:07.332118 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:07.332481 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 7619
I20260807 03:14:07.332710 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:14:07.333199 17532 external_mini_cluster.cc:1428] Setting key 7ca552f52a8c19a07d3d452c302687b5
I20260807 03:14:07.345161 7688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b: Bootstrap starting.
I20260807 03:14:07.346937 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:07.334984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:50103 (local address 127.0.0.1:44831)
0807 03:14:07.335797 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:07.335811 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:07.335850 (+ 39us) server_negotiation.cc:407] Connection header received
0807 03:14:07.337947 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:07.337971 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:07.338430 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:07.338761 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:07.339340 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:07.340622 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:07.342559 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:07.343235 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:07.343360 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:07.344618 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:07.344625 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:07.344637 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:14:07.345133 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:07.345208 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:14:07.345236 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:07.345423 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:07.345758 (+ 335us) server_negotiation.cc:299] Negotiation successful
0807 03:14:07.345944 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":378,"threads_started":1}
I20260807 03:14:07.348330 7688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:07.349367 7688 log.cc:824] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b: Log is configured to *not* fsync() on all Append() calls
I20260807 03:14:07.350797 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:14:07.352413 7688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b: No bootstrap required, opened a new log
I20260807 03:14:07.362413 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:07.355919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:36281 (local address 127.0.0.1:44831)
0807 03:14:07.356057 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:07.356061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:07.356414 (+ 353us) server_negotiation.cc:407] Connection header received
0807 03:14:07.356583 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:07.356587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:07.356644 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:07.356729 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:07.357280 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:07.358023 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:07.359890 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:07.360345 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:07.360446 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:07.361587 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:07.361592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:07.361595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:14:07.361611 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:07.361663 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:14:07.361678 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:07.361764 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:07.361988 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 03:14:07.362027 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260807 03:14:07.362686 7688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8953b28228f149ceb07df49e6e36843b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260807 03:14:07.363066 7688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:14:07.363194 7688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8953b28228f149ceb07df49e6e36843b, State: Initialized, Role: FOLLOWER
I20260807 03:14:07.363708 7688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8953b28228f149ceb07df49e6e36843b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260807 03:14:07.363972 7688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:14:07.364113 7688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:14:07.364284 7688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:07.366422 7688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8953b28228f149ceb07df49e6e36843b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260807 03:14:07.366940 7688 leader_election.cc:302] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8953b28228f149ceb07df49e6e36843b; no voters:
I20260807 03:14:07.367460 7688 leader_election.cc:288] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:14:07.367584 7698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:14:07.368319 7698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [term 1 LEADER]: Becoming Leader. State: Replica: 8953b28228f149ceb07df49e6e36843b, State: Running, Role: LEADER
I20260807 03:14:07.368883 7698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8953b28228f149ceb07df49e6e36843b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } }
I20260807 03:14:07.369503 7688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:14:07.372049 7699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8953b28228f149ceb07df49e6e36843b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8953b28228f149ceb07df49e6e36843b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } } }
I20260807 03:14:07.372077 7700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8953b28228f149ceb07df49e6e36843b. Latest consensus state: current_term: 1 leader_uuid: "8953b28228f149ceb07df49e6e36843b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8953b28228f149ceb07df49e6e36843b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44831 } } }
I20260807 03:14:07.372457 7699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:07.372483 7700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:07.374085 7706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:14:07.378679 7706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:14:07.388098 7706 catalog_manager.cc:1347] Generated new cluster ID: 939240b0d1df48dd8c20c9d09f34a3da
I20260807 03:14:07.388211 7706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:14:07.406198 7706 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:14:07.407428 7706 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:14:07.416951 7706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8953b28228f149ceb07df49e6e36843b: Generated new TSK 0
I20260807 03:14:07.417948 7706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:14:07.431782 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 7619
W20260807 03:14:07.432951 7683 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.78:50103
W20260807 03:14:07.433712 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 7619: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:14:07.433872 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 7619
W20260807 03:14:07.456687 7604 connection.cc:535] client connection to 127.0.0.1:44831 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44831): Connection reset by peer (error 104)
2026-08-07T03:14:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1930 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:14:07 dist-test-slave-tm1g krb5kdc[7720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:14:07 dist-test-slave-tm1g krb5kdc[7720](info): set up 2 sockets
Aug 07 03:14:07 dist-test-slave-tm1g krb5kdc[7720](info): commencing operation
krb5kdc: starting...
W20260807 03:14:08.569231 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.002s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:14:08 dist-test-slave-tm1g krb5kdc[7720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:14:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:14:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 03:14:08.851229 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:14:09.182520 7734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:09.182837 7734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:09.183013 7734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:14:09.190016 7734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:09.190162 7734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:14:09.190277 7734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:09.190407 7734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:09.190475 7734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:09.199931 7734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44055
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:09.201171 7734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:09.202523 7734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:14:09.214599 7739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:09.214542 7742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:09.214668 7740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:09.215685 7734 server_base.cc:1034] running on GCE node
I20260807 03:14:09.216493 7734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:09.217788 7734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:09.219029 7734 hybrid_clock.cc:648] HybridClock initialized: now 1786072449218996 us; error 49 us; skew 500 ppm
I20260807 03:14:09.219372 7734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:09 dist-test-slave-tm1g krb5kdc[7720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:09.227833 7734 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 03:14:09.229348 7734 webserver.cc:460] Webserver started at http://127.0.0.1:33543/ using document root <none> and password file <none>
I20260807 03:14:09.230031 7734 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:09.230181 7734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:09.230556 7734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:09.233428 7734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "eac0e147b67a47acbdc4c8a280ee5792"
format_stamp: "Formatted at 2026-08-07 03:14:09 on dist-test-slave-tm1g"
server_key: "1f8df8bbd7e154f61eac403266c24d61"
server_key_iv: "c9702369e1e0efd4b60a212ac80dbce0"
server_key_version: "encryptionkey@0"
I20260807 03:14:09.234211 7734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "eac0e147b67a47acbdc4c8a280ee5792"
format_stamp: "Formatted at 2026-08-07 03:14:09 on dist-test-slave-tm1g"
server_key: "1f8df8bbd7e154f61eac403266c24d61"
server_key_iv: "c9702369e1e0efd4b60a212ac80dbce0"
server_key_version: "encryptionkey@0"
I20260807 03:14:09.239238 7734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 03:14:09.242590 7749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:09.243839 7734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:14:09.244035 7734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "eac0e147b67a47acbdc4c8a280ee5792"
format_stamp: "Formatted at 2026-08-07 03:14:09 on dist-test-slave-tm1g"
server_key: "1f8df8bbd7e154f61eac403266c24d61"
server_key_iv: "c9702369e1e0efd4b60a212ac80dbce0"
server_key_version: "encryptionkey@0"
I20260807 03:14:09.244239 7734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:09.249815 7734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:09.253502 7734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:09.253785 7734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:09.272087 7734 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44055
I20260807 03:14:09.272133 7802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44055 every 8 connection(s)
I20260807 03:14:09.273418 7734 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:14:09.277024 7803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:09.278343 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 7734
I20260807 03:14:09.278546 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:14:09.279093 17532 external_mini_cluster.cc:1428] Setting key 35a7d291fdcb7edc34866a184ce8674b
I20260807 03:14:09.286653 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:09.280926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:55925 (local address 127.0.0.1:44055)
0807 03:14:09.281733 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:09.281744 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:09.281786 (+ 42us) server_negotiation.cc:407] Connection header received
0807 03:14:09.283412 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:09.283435 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:09.283738 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:09.284059 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:09.285091 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:09.285097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:09.285108 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:14:09.285122 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:09.285177 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:14:09.285200 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:09.285388 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:09.285494 (+ 106us) server_negotiation.cc:299] Negotiation successful
0807 03:14:09.285706 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":368,"threads_started":1}
I20260807 03:14:09.290279 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:14:09.290825 7803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792: Bootstrap starting.
I20260807 03:14:09.293785 7803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:09.295063 7803 log.cc:824] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792: Log is configured to *not* fsync() on all Append() calls
I20260807 03:14:09.297478 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:09.294784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:42643 (local address 127.0.0.1:44055)
0807 03:14:09.295008 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:09.295017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:09.295516 (+ 499us) server_negotiation.cc:407] Connection header received
0807 03:14:09.295663 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:09.295666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:09.295710 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:09.295784 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:09.296829 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:09.296833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:09.296836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:14:09.296849 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:09.296884 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:14:09.296898 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:09.296981 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:09.297126 (+ 145us) server_negotiation.cc:299] Negotiation successful
0807 03:14:09.297170 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260807 03:14:09.298514 7803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792: No bootstrap required, opened a new log
I20260807 03:14:09.305384 7803 raft_consensus.cc:348] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac0e147b67a47acbdc4c8a280ee5792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } }
I20260807 03:14:09.305745 7803 raft_consensus.cc:374] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:14:09.305853 7803 raft_consensus.cc:729] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac0e147b67a47acbdc4c8a280ee5792, State: Initialized, Role: FOLLOWER
I20260807 03:14:09.306496 7803 consensus_queue.cc:260] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac0e147b67a47acbdc4c8a280ee5792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } }
I20260807 03:14:09.306741 7803 raft_consensus.cc:388] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:14:09.306898 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:14:09.307078 7803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:09.309141 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac0e147b67a47acbdc4c8a280ee5792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } }
I20260807 03:14:09.309604 7803 leader_election.cc:302] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eac0e147b67a47acbdc4c8a280ee5792; no voters:
I20260807 03:14:09.310144 7803 leader_election.cc:288] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:14:09.310298 7813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:14:09.311058 7813 raft_consensus.cc:686] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [term 1 LEADER]: Becoming Leader. State: Replica: eac0e147b67a47acbdc4c8a280ee5792, State: Running, Role: LEADER
I20260807 03:14:09.311537 7813 consensus_queue.cc:237] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac0e147b67a47acbdc4c8a280ee5792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } }
I20260807 03:14:09.312247 7803 sys_catalog.cc:564] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:14:09.314500 7815 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eac0e147b67a47acbdc4c8a280ee5792. Latest consensus state: current_term: 1 leader_uuid: "eac0e147b67a47acbdc4c8a280ee5792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac0e147b67a47acbdc4c8a280ee5792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } } }
I20260807 03:14:09.314842 7815 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:09.316478 7819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:14:09.316502 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eac0e147b67a47acbdc4c8a280ee5792" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac0e147b67a47acbdc4c8a280ee5792" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44055 } } }
I20260807 03:14:09.316789 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:09.321475 7819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:14:09.330956 7819 catalog_manager.cc:1347] Generated new cluster ID: c171d8d95b7a49938770a048095819c7
I20260807 03:14:09.331067 7819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:14:09.404074 7819 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:14:09.404552 7819 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:14:09.413847 7819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eac0e147b67a47acbdc4c8a280ee5792: Generated new TSK 0
I20260807 03:14:09.414680 7819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:14:09.427489 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 7734
W20260807 03:14:09.428534 7798 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.78:55925
W20260807 03:14:09.429219 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 7734: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:14:09.429378 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 7734
2026-08-07T03:14:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1988 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:14:09 dist-test-slave-tm1g krb5kdc[7834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:14:09 dist-test-slave-tm1g krb5kdc[7834](info): set up 2 sockets
Aug 07 03:14:09 dist-test-slave-tm1g krb5kdc[7834](info): commencing operation
krb5kdc: starting...
W20260807 03:14:10.551875 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.004s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:14:10 dist-test-slave-tm1g krb5kdc[7834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:14:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:14:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:14:10.809708 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:34051
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:34635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:34051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:14:11.133415 7847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:11.133728 7847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:11.133894 7847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:14:11.140978 7847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:11.141129 7847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:14:11.141240 7847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:11.141336 7847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:11.141407 7847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:11.151371 7847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:34635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:34051
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:34051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:11.152614 7847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:11.153913 7847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:14:11.165555 7852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:11.165616 7853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:11.165910 7855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:11.166682 7847 server_base.cc:1034] running on GCE node
I20260807 03:14:11.167341 7847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:11.168545 7847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:11.169786 7847 hybrid_clock.cc:648] HybridClock initialized: now 1786072451169745 us; error 59 us; skew 500 ppm
I20260807 03:14:11.170135 7847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:11 dist-test-slave-tm1g krb5kdc[7834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072451, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:11.173272 7847 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:14:11.174800 7847 webserver.cc:460] Webserver started at http://127.17.31.62:35591/ using document root <none> and password file <none>
I20260807 03:14:11.175467 7847 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:11.175622 7847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:11.175926 7847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:11.178893 7847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "e007ece1d8534a22a4371dfe7be86bc1"
format_stamp: "Formatted at 2026-08-07 03:14:11 on dist-test-slave-tm1g"
server_key: "743974fde5c81fb27e921c9e1b746c99"
server_key_iv: "c97daeddfdea05a5460cfbeab5dc316c"
server_key_version: "encryptionkey@0"
I20260807 03:14:11.179651 7847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "e007ece1d8534a22a4371dfe7be86bc1"
format_stamp: "Formatted at 2026-08-07 03:14:11 on dist-test-slave-tm1g"
server_key: "743974fde5c81fb27e921c9e1b746c99"
server_key_iv: "c97daeddfdea05a5460cfbeab5dc316c"
server_key_version: "encryptionkey@0"
I20260807 03:14:11.184514 7847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 03:14:11.187827 7862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:11.188989 7847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 03:14:11.189188 7847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "e007ece1d8534a22a4371dfe7be86bc1"
format_stamp: "Formatted at 2026-08-07 03:14:11 on dist-test-slave-tm1g"
server_key: "743974fde5c81fb27e921c9e1b746c99"
server_key_iv: "c97daeddfdea05a5460cfbeab5dc316c"
server_key_version: "encryptionkey@0"
I20260807 03:14:11.189385 7847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:11.215121 7847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:11.219352 7847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:11.219640 7847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:11.238173 7847 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:34051
I20260807 03:14:11.238217 7915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:34051 every 8 connection(s)
I20260807 03:14:11.239488 7847 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:14:11.243001 7916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:11.244266 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 7847
I20260807 03:14:11.244441 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:14:11.244922 17532 external_mini_cluster.cc:1428] Setting key 5e135ed7cfe2359854b836b4315e46b3
I20260807 03:14:11.256990 7916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1: Bootstrap starting.
Aug 07 03:14:11 dist-test-slave-tm1g krb5kdc[7834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:14:11.260200 7916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:11.261355 7916 log.cc:824] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1: Log is configured to *not* fsync() on all Append() calls
I20260807 03:14:11.264245 7916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1: No bootstrap required, opened a new log
I20260807 03:14:11.268082 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:11.246760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51509 (local address 127.17.31.62:34051)
0807 03:14:11.247544 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:11.247557 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:11.247608 (+ 51us) server_negotiation.cc:407] Connection header received
0807 03:14:11.249412 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:11.249449 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:11.249980 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:11.250436 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:11.251697 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:11.253759 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:11.255069 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:11.256111 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:11.256294 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:11.260801 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:11.260808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:11.260821 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:11.260875 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:11.263872 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:11.264781 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:11.264787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:11.264793 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:11.264973 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:11.265488 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:11.265491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:11.265493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:11.265974 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:11.266216 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:11.266795 (+ 579us) server_negotiation.cc:299] Negotiation successful
0807 03:14:11.267118 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":321,"threads_started":1}
I20260807 03:14:11.271848 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:14:11.273192 7916 raft_consensus.cc:348] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e007ece1d8534a22a4371dfe7be86bc1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34051 } }
I20260807 03:14:11.273615 7916 raft_consensus.cc:374] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:14:11.273713 7916 raft_consensus.cc:729] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e007ece1d8534a22a4371dfe7be86bc1, State: Initialized, Role: FOLLOWER
I20260807 03:14:11.274293 7916 consensus_queue.cc:260] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e007ece1d8534a22a4371dfe7be86bc1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34051 } }
I20260807 03:14:11.274621 7916 raft_consensus.cc:388] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:14:11.274757 7916 raft_consensus.cc:482] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:14:11.274940 7916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:11.278156 7916 raft_consensus.cc:504] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e007ece1d8534a22a4371dfe7be86bc1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34051 } }
I20260807 03:14:11.278739 7916 leader_election.cc:302] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e007ece1d8534a22a4371dfe7be86bc1; no voters:
I20260807 03:14:11.279374 7916 leader_election.cc:288] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:14:11.279433 7926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:14:11.280534 7926 raft_consensus.cc:686] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [term 1 LEADER]: Becoming Leader. State: Replica: e007ece1d8534a22a4371dfe7be86bc1, State: Running, Role: LEADER
I20260807 03:14:11.281102 7926 consensus_queue.cc:237] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e007ece1d8534a22a4371dfe7be86bc1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34051 } }
I20260807 03:14:11.281560 7916 sys_catalog.cc:564] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:14:11.284224 7927 sys_catalog.cc:455] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e007ece1d8534a22a4371dfe7be86bc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e007ece1d8534a22a4371dfe7be86bc1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34051 } } }
I20260807 03:14:11.284675 7927 sys_catalog.cc:458] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:11.287572 7928 sys_catalog.cc:455] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e007ece1d8534a22a4371dfe7be86bc1. Latest consensus state: current_term: 1 leader_uuid: "e007ece1d8534a22a4371dfe7be86bc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e007ece1d8534a22a4371dfe7be86bc1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 34051 } } }
I20260807 03:14:11.287928 7928 sys_catalog.cc:458] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:11.288157 7931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:14:11.293969 7931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:14:11.295885 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:11.277071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41947 (local address 127.17.31.62:34051)
0807 03:14:11.277281 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:11.277289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:11.277679 (+ 390us) server_negotiation.cc:407] Connection header received
0807 03:14:11.277831 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:11.277835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:11.277886 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:11.277959 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:11.279937 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:11.281147 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:11.283034 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:11.283732 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:11.283861 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:11.287603 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:11.287613 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:11.287620 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:11.287666 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:11.291843 (+ 4177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:11.292807 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:11.292812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:11.292815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:11.292950 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:11.293547 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:11.293552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:11.293555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:11.293972 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:11.294219 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:11.295358 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0807 03:14:11.295529 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 03:14:11.304147 7931 catalog_manager.cc:1347] Generated new cluster ID: 242c633bc1ee4df4befd775179b04af7
I20260807 03:14:11.304250 7931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:14:11.318686 7931 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:14:11.320436 7931 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:14:11.329392 7931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e007ece1d8534a22a4371dfe7be86bc1: Generated new TSK 0
I20260807 03:14:11.330139 7931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:14:11.338567 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 7847
I20260807 03:14:11.339712 7911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:14:11.428130 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 7847
W20260807 03:14:11.445631 7832 connection.cc:535] client connection to 127.17.31.62:34051 recv error: Network error: failed to read from TLS socket (remote: 127.17.31.62:34051): Connection reset by peer (error 104)
2026-08-07T03:14:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:14:11 dist-test-slave-tm1g krb5kdc[7948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:14:11 dist-test-slave-tm1g krb5kdc[7948](info): set up 2 sockets
Aug 07 03:14:11 dist-test-slave-tm1g krb5kdc[7948](info): commencing operation
krb5kdc: starting...
W20260807 03:14:12.551944 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:14:12 dist-test-slave-tm1g krb5kdc[7948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:14:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:14:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:14:12.817452 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:38553
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:38187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:38553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:14:13.144760 7962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:13.145076 7962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:13.145251 7962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:14:13.152258 7962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:13.152423 7962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:14:13.152540 7962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:13.152654 7962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:13.152746 7962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:13.161862 7962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:38187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:38553
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:38553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:13.163100 7962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:13.164417 7962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:14:13.175776 7970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:13.175829 7968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:13.175935 7967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:13.177003 7962 server_base.cc:1034] running on GCE node
I20260807 03:14:13.177727 7962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:13.179091 7962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:13.180336 7962 hybrid_clock.cc:648] HybridClock initialized: now 1786072453180307 us; error 45 us; skew 500 ppm
I20260807 03:14:13.180670 7962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:13 dist-test-slave-tm1g krb5kdc[7948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072453, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:13.183760 7962 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:14:13.185063 7962 webserver.cc:460] Webserver started at http://127.17.31.62:46565/ using document root <none> and password file <none>
I20260807 03:14:13.185748 7962 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:13.185897 7962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:13.186200 7962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:13.189067 7962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "e95cbdc85993411fb130da64d76c3574"
format_stamp: "Formatted at 2026-08-07 03:14:13 on dist-test-slave-tm1g"
server_key: "b0b6a1a610affe20490817ef148a078c"
server_key_iv: "2788a96a61ea12abc3f3883c19c524c8"
server_key_version: "encryptionkey@0"
I20260807 03:14:13.189852 7962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "e95cbdc85993411fb130da64d76c3574"
format_stamp: "Formatted at 2026-08-07 03:14:13 on dist-test-slave-tm1g"
server_key: "b0b6a1a610affe20490817ef148a078c"
server_key_iv: "2788a96a61ea12abc3f3883c19c524c8"
server_key_version: "encryptionkey@0"
I20260807 03:14:13.194713 7962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260807 03:14:13.197961 7977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:13.199208 7962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:14:13.199383 7962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "e95cbdc85993411fb130da64d76c3574"
format_stamp: "Formatted at 2026-08-07 03:14:13 on dist-test-slave-tm1g"
server_key: "b0b6a1a610affe20490817ef148a078c"
server_key_iv: "2788a96a61ea12abc3f3883c19c524c8"
server_key_version: "encryptionkey@0"
I20260807 03:14:13.199568 7962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:13.225543 7962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:13.229766 7962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:13.230058 7962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:13.248159 7962 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:38553
I20260807 03:14:13.248202 8029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:38553 every 8 connection(s)
I20260807 03:14:13.249464 7962 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:14:13.252776 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 7962
I20260807 03:14:13.252970 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:14:13.253216 8030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:13.253509 17532 external_mini_cluster.cc:1428] Setting key 9a9c8b8c3a85d40a63223dc53ea02da6
I20260807 03:14:13.268159 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:13.255297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46499 (local address 127.17.31.62:38553)
0807 03:14:13.256149 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:13.256162 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:13.256217 (+ 55us) server_negotiation.cc:407] Connection header received
0807 03:14:13.258209 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:13.258238 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:13.258772 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:13.259196 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:13.259709 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:13.261738 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:13.263113 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:13.264160 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:13.264336 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:13.265502 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:13.265513 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:13.265531 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:14:13.266242 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:13.266352 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:14:13.266382 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:13.266531 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:13.266788 (+ 257us) server_negotiation.cc:299] Negotiation successful
0807 03:14:13.266947 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":365,"threads_started":1}
I20260807 03:14:13.268448 8030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574: Bootstrap starting.
I20260807 03:14:13.272150 8030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:13.272217 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:14:13.273149 8030 log.cc:824] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574: Log is configured to *not* fsync() on all Append() calls
I20260807 03:14:13.275976 8030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574: No bootstrap required, opened a new log
I20260807 03:14:13.283962 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:13.277020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37297 (local address 127.17.31.62:38553)
0807 03:14:13.277242 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:13.277250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:13.277802 (+ 552us) server_negotiation.cc:407] Connection header received
0807 03:14:13.277934 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:13.277940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:13.278013 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:13.278174 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:13.278733 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:13.279842 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:13.281134 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:13.281837 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:13.281997 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:13.283141 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:13.283147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:13.283151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:14:13.283171 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:13.283230 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:14:13.283256 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:13.283373 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:13.283569 (+ 196us) server_negotiation.cc:299] Negotiation successful
0807 03:14:13.283623 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 03:14:13.285255 8030 raft_consensus.cc:348] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95cbdc85993411fb130da64d76c3574" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 38553 } }
I20260807 03:14:13.285773 8030 raft_consensus.cc:374] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:14:13.285907 8030 raft_consensus.cc:729] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e95cbdc85993411fb130da64d76c3574, State: Initialized, Role: FOLLOWER
I20260807 03:14:13.286532 8030 consensus_queue.cc:260] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95cbdc85993411fb130da64d76c3574" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 38553 } }
I20260807 03:14:13.286778 8030 raft_consensus.cc:388] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:14:13.286940 8030 raft_consensus.cc:482] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:14:13.287115 8030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:13.289279 8030 raft_consensus.cc:504] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95cbdc85993411fb130da64d76c3574" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 38553 } }
I20260807 03:14:13.289776 8030 leader_election.cc:302] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e95cbdc85993411fb130da64d76c3574; no voters:
I20260807 03:14:13.290498 8030 leader_election.cc:288] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:14:13.290611 8040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:14:13.291412 8040 raft_consensus.cc:686] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [term 1 LEADER]: Becoming Leader. State: Replica: e95cbdc85993411fb130da64d76c3574, State: Running, Role: LEADER
I20260807 03:14:13.291939 8040 consensus_queue.cc:237] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95cbdc85993411fb130da64d76c3574" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 38553 } }
I20260807 03:14:13.292591 8030 sys_catalog.cc:564] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:14:13.294700 8042 sys_catalog.cc:455] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e95cbdc85993411fb130da64d76c3574. Latest consensus state: current_term: 1 leader_uuid: "e95cbdc85993411fb130da64d76c3574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95cbdc85993411fb130da64d76c3574" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 38553 } } }
I20260807 03:14:13.295073 8042 sys_catalog.cc:458] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:13.296823 8041 sys_catalog.cc:455] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e95cbdc85993411fb130da64d76c3574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e95cbdc85993411fb130da64d76c3574" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 38553 } } }
I20260807 03:14:13.297190 8041 sys_catalog.cc:458] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:13.297361 8045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:14:13.305488 8045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:14:13.314538 8045 catalog_manager.cc:1347] Generated new cluster ID: b1bcd59ee6fc47ffa38990cf84263fe8
I20260807 03:14:13.314666 8045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:14:13.331653 8045 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:14:13.333154 8045 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:14:13.339967 8045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e95cbdc85993411fb130da64d76c3574: Generated new TSK 0
I20260807 03:14:13.340716 8045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:14:13.345991 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 7962
W20260807 03:14:13.347132 8025 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46499
W20260807 03:14:13.347869 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 7962: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:14:13.348042 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 7962
W20260807 03:14:13.365950 7946 connection.cc:535] client connection to 127.17.31.62:38553 recv error: Network error: failed to read from TLS socket (remote: 127.17.31.62:38553): Connection reset by peer (error 104)
2026-08-07T03:14:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1920 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:14:13 dist-test-slave-tm1g krb5kdc[8061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:14:13 dist-test-slave-tm1g krb5kdc[8061](info): set up 2 sockets
Aug 07 03:14:13 dist-test-slave-tm1g krb5kdc[8061](info): commencing operation
krb5kdc: starting...
W20260807 03:14:14.472028 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:14:14 dist-test-slave-tm1g krb5kdc[8061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:14:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:14:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:14:14.730515 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:43413
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:46065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:43413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:14:15.059402 8074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:15.059656 8074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:15.059748 8074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:14:15.066617 8074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:15.066701 8074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:14:15.066744 8074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:15.066802 8074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:15.066839 8074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:15.075870 8074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:46065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:43413
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:43413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:15.076920 8074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:15.078150 8074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:14:15.089798 8079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:15.089924 8082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:15.090332 8080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:15.090806 8074 server_base.cc:1034] running on GCE node
I20260807 03:14:15.091495 8074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:15.092798 8074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:15.094069 8074 hybrid_clock.cc:648] HybridClock initialized: now 1786072455094036 us; error 48 us; skew 500 ppm
I20260807 03:14:15.094486 8074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:15 dist-test-slave-tm1g krb5kdc[8061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072455, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:15.102623 8074 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:14:15.104215 8074 webserver.cc:460] Webserver started at http://127.17.31.62:40131/ using document root <none> and password file <none>
I20260807 03:14:15.104941 8074 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:15.105091 8074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:15.105438 8074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:15.108533 8074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "9bd7f5b6ff66468caa2e7db7c1aea39f"
format_stamp: "Formatted at 2026-08-07 03:14:15 on dist-test-slave-tm1g"
server_key: "eee4e3248a630dc6da4be6e41354ed75"
server_key_iv: "bf305c43087962b8909830afef7592cf"
server_key_version: "encryptionkey@0"
I20260807 03:14:15.109413 8074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "9bd7f5b6ff66468caa2e7db7c1aea39f"
format_stamp: "Formatted at 2026-08-07 03:14:15 on dist-test-slave-tm1g"
server_key: "eee4e3248a630dc6da4be6e41354ed75"
server_key_iv: "bf305c43087962b8909830afef7592cf"
server_key_version: "encryptionkey@0"
I20260807 03:14:15.115700 8074 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260807 03:14:15.119128 8089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:15.120476 8074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:14:15.120671 8074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "9bd7f5b6ff66468caa2e7db7c1aea39f"
format_stamp: "Formatted at 2026-08-07 03:14:15 on dist-test-slave-tm1g"
server_key: "eee4e3248a630dc6da4be6e41354ed75"
server_key_iv: "bf305c43087962b8909830afef7592cf"
server_key_version: "encryptionkey@0"
I20260807 03:14:15.120872 8074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:15.126721 8074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:15.130713 8074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:15.131006 8074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:15.149777 8074 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:43413
I20260807 03:14:15.149819 8141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:43413 every 8 connection(s)
I20260807 03:14:15.151140 8074 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:14:15.154590 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 8074
I20260807 03:14:15.154799 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:14:15.155097 8142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:15.155383 17532 external_mini_cluster.cc:1428] Setting key c4cec90ea04927ecf061ccce397ec75f
I20260807 03:14:15.163254 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:15.157222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59257 (local address 127.17.31.62:43413)
0807 03:14:15.157988 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:15.157999 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:15.158040 (+ 41us) server_negotiation.cc:407] Connection header received
0807 03:14:15.159887 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:15.159909 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:15.160229 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:15.160622 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:15.161677 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:15.161686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:15.161697 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:14:15.161710 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:15.161766 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:14:15.161787 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:15.162063 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:15.162220 (+ 157us) server_negotiation.cc:299] Negotiation successful
0807 03:14:15.162424 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":333,"threads_started":1}
I20260807 03:14:15.166826 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:14:15.168118 8142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f: Bootstrap starting.
I20260807 03:14:15.171824 8142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:15.173218 8142 log.cc:824] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f: Log is configured to *not* fsync() on all Append() calls
I20260807 03:14:15.174013 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:15.171208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58145 (local address 127.17.31.62:43413)
0807 03:14:15.171378 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:15.171383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:15.171926 (+ 543us) server_negotiation.cc:407] Connection header received
0807 03:14:15.172117 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:15.172122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:15.172193 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:15.172326 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:15.173200 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:15.173212 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:15.173217 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:14:15.173236 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:15.173284 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:14:15.173307 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:15.173424 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:15.173592 (+ 168us) server_negotiation.cc:299] Negotiation successful
0807 03:14:15.173648 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:14:15.176292 8142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f: No bootstrap required, opened a new log
I20260807 03:14:15.183952 8142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd7f5b6ff66468caa2e7db7c1aea39f" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 43413 } }
I20260807 03:14:15.184266 8142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:14:15.184337 8142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd7f5b6ff66468caa2e7db7c1aea39f, State: Initialized, Role: FOLLOWER
I20260807 03:14:15.184799 8142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd7f5b6ff66468caa2e7db7c1aea39f" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 43413 } }
I20260807 03:14:15.185001 8142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:14:15.185098 8142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:14:15.185214 8142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:15.187337 8142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd7f5b6ff66468caa2e7db7c1aea39f" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 43413 } }
I20260807 03:14:15.187749 8142 leader_election.cc:302] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bd7f5b6ff66468caa2e7db7c1aea39f; no voters:
I20260807 03:14:15.188253 8142 leader_election.cc:288] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:14:15.188403 8152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:14:15.189030 8152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [term 1 LEADER]: Becoming Leader. State: Replica: 9bd7f5b6ff66468caa2e7db7c1aea39f, State: Running, Role: LEADER
I20260807 03:14:15.189603 8152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd7f5b6ff66468caa2e7db7c1aea39f" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 43413 } }
I20260807 03:14:15.190379 8142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:14:15.192348 8153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bd7f5b6ff66468caa2e7db7c1aea39f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd7f5b6ff66468caa2e7db7c1aea39f" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 43413 } } }
I20260807 03:14:15.192780 8153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:15.192668 8154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bd7f5b6ff66468caa2e7db7c1aea39f. Latest consensus state: current_term: 1 leader_uuid: "9bd7f5b6ff66468caa2e7db7c1aea39f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd7f5b6ff66468caa2e7db7c1aea39f" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 43413 } } }
I20260807 03:14:15.193030 8154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:15.195053 8159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:14:15.199577 8159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:14:15.200326 8169 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 03:14:15.208249 8159 catalog_manager.cc:1347] Generated new cluster ID: c2c42d91b72d4511b109f0936b6723e6
I20260807 03:14:15.208381 8159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:14:15.228716 8159 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:14:15.229117 8159 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:14:15.237346 8159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bd7f5b6ff66468caa2e7db7c1aea39f: Generated new TSK 0
I20260807 03:14:15.238049 8159 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:14:15.252137 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 8074
W20260807 03:14:15.253136 8137 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59257
W20260807 03:14:15.253836 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 8074: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:14:15.254004 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 8074
2026-08-07T03:14:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1905 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:14:15 dist-test-slave-tm1g krb5kdc[8173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:14:15 dist-test-slave-tm1g krb5kdc[8173](info): set up 2 sockets
Aug 07 03:14:15 dist-test-slave-tm1g krb5kdc[8173](info): commencing operation
krb5kdc: starting...
W20260807 03:14:16.379513 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.003s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:14:16 dist-test-slave-tm1g krb5kdc[8173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:14:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:14:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:14:16.646262 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:41269
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:41269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:14:16.973434 8186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:16.973781 8186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:16.973959 8186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:14:16.981034 8186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:16.981185 8186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:14:16.981315 8186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:16.981439 8186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:16.981534 8186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:16.991771 8186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:41269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:41269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:16.993070 8186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:16.994462 8186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:14:17.006722 8192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:17.006632 8195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:17.006793 8193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:17.007740 8186 server_base.cc:1034] running on GCE node
I20260807 03:14:17.008491 8186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:17.009809 8186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:17.011053 8186 hybrid_clock.cc:648] HybridClock initialized: now 1786072457011018 us; error 30 us; skew 500 ppm
I20260807 03:14:17.011408 8186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:17 dist-test-slave-tm1g krb5kdc[8173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072457, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:17.014524 8186 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:14:17.015996 8186 webserver.cc:460] Webserver started at http://127.17.31.62:38935/ using document root <none> and password file <none>
I20260807 03:14:17.016686 8186 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:17.016842 8186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:17.017146 8186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:17.020143 8186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "58627444983747c5a59e69f109765f97"
format_stamp: "Formatted at 2026-08-07 03:14:17 on dist-test-slave-tm1g"
server_key: "481607e846399031463772c609de0736"
server_key_iv: "b32539e445ff4e02a998e127affd089d"
server_key_version: "encryptionkey@0"
I20260807 03:14:17.020905 8186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "58627444983747c5a59e69f109765f97"
format_stamp: "Formatted at 2026-08-07 03:14:17 on dist-test-slave-tm1g"
server_key: "481607e846399031463772c609de0736"
server_key_iv: "b32539e445ff4e02a998e127affd089d"
server_key_version: "encryptionkey@0"
I20260807 03:14:17.026001 8186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 03:14:17.029390 8202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:17.030714 8186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:14:17.030906 8186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "58627444983747c5a59e69f109765f97"
format_stamp: "Formatted at 2026-08-07 03:14:17 on dist-test-slave-tm1g"
server_key: "481607e846399031463772c609de0736"
server_key_iv: "b32539e445ff4e02a998e127affd089d"
server_key_version: "encryptionkey@0"
I20260807 03:14:17.031105 8186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:17.061703 8186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:17.065639 8186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:17.065939 8186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:17.084703 8186 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:41269
I20260807 03:14:17.084740 8255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:41269 every 8 connection(s)
I20260807 03:14:17.086047 8186 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:14:17.089799 8256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:17.090687 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 8186
I20260807 03:14:17.090911 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:14:17.091363 17532 external_mini_cluster.cc:1428] Setting key 623c2dc26c13ba1b6c1d58ec23f42d1c
I20260807 03:14:17.103482 8256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97: Bootstrap starting.
Aug 07 03:14:17 dist-test-slave-tm1g krb5kdc[8173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:14:17.107270 8256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:17.108477 8256 log.cc:824] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97: Log is configured to *not* fsync() on all Append() calls
I20260807 03:14:17.111431 8256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97: No bootstrap required, opened a new log
I20260807 03:14:17.114933 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:17.093192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47925 (local address 127.17.31.62:41269)
0807 03:14:17.094042 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:17.094053 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:17.094092 (+ 39us) server_negotiation.cc:407] Connection header received
0807 03:14:17.095761 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:17.095786 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:17.096170 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:17.096483 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:17.098293 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:17.099640 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:17.101531 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:17.102183 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:17.102394 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:17.107225 (+ 4831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:17.107232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:17.107245 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:17.107288 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:17.110807 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:17.111508 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:17.111515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:17.111521 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:17.111715 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:17.112120 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:17.112125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:17.112129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:17.112616 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:17.112958 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:17.113386 (+ 428us) server_negotiation.cc:299] Negotiation successful
0807 03:14:17.113870 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":385,"threads_started":1}
I20260807 03:14:17.118539 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:14:17.119871 8256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58627444983747c5a59e69f109765f97" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41269 } }
I20260807 03:14:17.120282 8256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:14:17.120406 8256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58627444983747c5a59e69f109765f97, State: Initialized, Role: FOLLOWER
I20260807 03:14:17.121027 8256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58627444983747c5a59e69f109765f97" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41269 } }
I20260807 03:14:17.121326 8256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:14:17.121460 8256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:14:17.121626 8256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:17.124179 8256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58627444983747c5a59e69f109765f97" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41269 } }
I20260807 03:14:17.124804 8256 leader_election.cc:302] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58627444983747c5a59e69f109765f97; no voters:
I20260807 03:14:17.125502 8256 leader_election.cc:288] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:14:17.125597 8266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:14:17.127255 8266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [term 1 LEADER]: Becoming Leader. State: Replica: 58627444983747c5a59e69f109765f97, State: Running, Role: LEADER
I20260807 03:14:17.128021 8256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:14:17.127895 8266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58627444983747c5a59e69f109765f97" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41269 } }
I20260807 03:14:17.132252 8267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58627444983747c5a59e69f109765f97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58627444983747c5a59e69f109765f97" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41269 } } }
I20260807 03:14:17.132690 8267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:17.133319 8268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58627444983747c5a59e69f109765f97. Latest consensus state: current_term: 1 leader_uuid: "58627444983747c5a59e69f109765f97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58627444983747c5a59e69f109765f97" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 41269 } } }
I20260807 03:14:17.133659 8268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:17.139914 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:17.124094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46445 (local address 127.17.31.62:41269)
0807 03:14:17.124292 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:17.124299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:17.124764 (+ 465us) server_negotiation.cc:407] Connection header received
0807 03:14:17.124986 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:17.124991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:17.125063 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:17.125169 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:17.127016 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:17.128074 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:17.129483 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:17.129930 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:17.130085 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:17.133334 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:17.133344 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:17.133352 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:17.133401 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:17.137169 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:17.137830 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:17.137837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:17.137841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:17.137988 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:17.138419 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:17.138425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:17.138428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:17.138719 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:17.138978 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:17.139335 (+ 357us) server_negotiation.cc:299] Negotiation successful
0807 03:14:17.139554 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 03:14:17.142841 8273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:14:17.148543 8273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:14:17.157078 8273 catalog_manager.cc:1347] Generated new cluster ID: 0a2c2e6a8d7446c29336d78afb3d05c4
I20260807 03:14:17.157193 8273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:14:17.170780 8273 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:14:17.172581 8273 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:14:17.183740 8273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58627444983747c5a59e69f109765f97: Generated new TSK 0
I20260807 03:14:17.184548 8273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:14:17.193285 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 8186
I20260807 03:14:17.194422 8251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:14:17.284963 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 8186
2026-08-07T03:14:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2031 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:14:17 dist-test-slave-tm1g krb5kdc[8290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:14:17 dist-test-slave-tm1g krb5kdc[8290](info): set up 2 sockets
Aug 07 03:14:17 dist-test-slave-tm1g krb5kdc[8290](info): commencing operation
krb5kdc: starting...
W20260807 03:14:18.440405 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:14:18 dist-test-slave-tm1g krb5kdc[8290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:14:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:14:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:14:18.697692 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:46383
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:43739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:46383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:14:19.022534 8305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:19.022804 8305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:19.022994 8305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:14:19.030169 8305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:19.030293 8305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:14:19.030405 8305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:19.030550 8305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:19.030613 8305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:19.039902 8305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:43739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:46383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:46383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:19.041074 8305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:19.042431 8305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:14:19.054281 8310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:19.054214 8311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:19.054333 8313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:19.055271 8305 server_base.cc:1034] running on GCE node
I20260807 03:14:19.056006 8305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:19.057340 8305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:19.058588 8305 hybrid_clock.cc:648] HybridClock initialized: now 1786072459058531 us; error 46 us; skew 500 ppm
I20260807 03:14:19.058929 8305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:19 dist-test-slave-tm1g krb5kdc[8290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072459, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:19.061895 8305 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:14:19.063253 8305 webserver.cc:460] Webserver started at http://127.17.31.62:34755/ using document root <none> and password file <none>
I20260807 03:14:19.063865 8305 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:19.063961 8305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:19.064200 8305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:19.067056 8305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "56a99619013648eba31e362522317281"
format_stamp: "Formatted at 2026-08-07 03:14:19 on dist-test-slave-tm1g"
server_key: "a80cc04e2fe3590d4629977023e5f4f5"
server_key_iv: "a726d435235a7ab0488474569dddbd1d"
server_key_version: "encryptionkey@0"
I20260807 03:14:19.067750 8305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "56a99619013648eba31e362522317281"
format_stamp: "Formatted at 2026-08-07 03:14:19 on dist-test-slave-tm1g"
server_key: "a80cc04e2fe3590d4629977023e5f4f5"
server_key_iv: "a726d435235a7ab0488474569dddbd1d"
server_key_version: "encryptionkey@0"
I20260807 03:14:19.072562 8305 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 03:14:19.075984 8320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:19.077224 8305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 03:14:19.077416 8305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "56a99619013648eba31e362522317281"
format_stamp: "Formatted at 2026-08-07 03:14:19 on dist-test-slave-tm1g"
server_key: "a80cc04e2fe3590d4629977023e5f4f5"
server_key_iv: "a726d435235a7ab0488474569dddbd1d"
server_key_version: "encryptionkey@0"
I20260807 03:14:19.077600 8305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:19.098356 8305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:19.102244 8305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:19.102556 8305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:19.121227 8305 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:46383
I20260807 03:14:19.121299 8373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:46383 every 8 connection(s)
I20260807 03:14:19.122581 8305 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:14:19.126052 8375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:19.131007 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 8305
I20260807 03:14:19.131196 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:14:19.131736 17532 external_mini_cluster.cc:1428] Setting key 8226ea6405c973276c03bd5a09cfdedf
I20260807 03:14:19.137836 8375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281: Bootstrap starting.
I20260807 03:14:19.140832 8375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:19.141830 8375 log.cc:824] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281: Log is configured to *not* fsync() on all Append() calls
I20260807 03:14:19.145145 8375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281: No bootstrap required, opened a new log
I20260807 03:14:19.145679 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:19.133732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36127 (local address 127.17.31.62:46383)
0807 03:14:19.134569 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:19.134580 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:19.134621 (+ 41us) server_negotiation.cc:407] Connection header received
0807 03:14:19.136564 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:19.136585 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:19.136926 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:19.137316 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:19.137872 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:19.139362 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:19.140701 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:19.141430 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:19.141674 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:19.142954 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:19.142964 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:19.142976 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:14:19.143695 (+ 719us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:19.143783 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:14:19.143814 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:19.143936 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:19.144158 (+ 222us) server_negotiation.cc:299] Negotiation successful
0807 03:14:19.144413 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":459,"threads_started":1}
I20260807 03:14:19.149348 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:14:19.154695 8375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a99619013648eba31e362522317281" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46383 } }
I20260807 03:14:19.155139 8375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:14:19.155304 8375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a99619013648eba31e362522317281, State: Initialized, Role: FOLLOWER
I20260807 03:14:19.155959 8375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a99619013648eba31e362522317281" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46383 } }
I20260807 03:14:19.156260 8375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:14:19.156412 8375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:14:19.156603 8375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:19.159646 8375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a99619013648eba31e362522317281" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46383 } }
I20260807 03:14:19.160211 8375 leader_election.cc:302] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56a99619013648eba31e362522317281; no voters:
I20260807 03:14:19.160861 8375 leader_election.cc:288] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:14:19.161031 8385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:14:19.161442 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:19.154395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59853 (local address 127.17.31.62:46383)
0807 03:14:19.154601 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:19.154609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:19.155042 (+ 433us) server_negotiation.cc:407] Connection header received
0807 03:14:19.155272 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:19.155279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:19.155348 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:19.155456 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:19.156061 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:19.156976 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:19.158876 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:19.159300 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:19.159390 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:19.160534 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:19.160539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:19.160542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:14:19.160558 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:19.160620 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:14:19.160651 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:19.160765 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:19.160951 (+ 186us) server_negotiation.cc:299] Negotiation successful
0807 03:14:19.161060 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 03:14:19.163014 8375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:14:19.168025 8385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [term 1 LEADER]: Becoming Leader. State: Replica: 56a99619013648eba31e362522317281, State: Running, Role: LEADER
I20260807 03:14:19.168470 8385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a99619013648eba31e362522317281" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46383 } }
I20260807 03:14:19.171293 8386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56a99619013648eba31e362522317281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a99619013648eba31e362522317281" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46383 } } }
I20260807 03:14:19.171693 8386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:19.173420 8399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:14:19.174813 8387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56a99619013648eba31e362522317281. Latest consensus state: current_term: 1 leader_uuid: "56a99619013648eba31e362522317281" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a99619013648eba31e362522317281" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46383 } } }
I20260807 03:14:19.175072 8387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:19.177964 8399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:14:19.178719 8403 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 03:14:19.186723 8399 catalog_manager.cc:1347] Generated new cluster ID: 9c350eaadeed47f6b1dcb2014b0574dd
I20260807 03:14:19.186839 8399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:14:19.195942 8399 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:14:19.197121 8399 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:14:19.202159 8399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56a99619013648eba31e362522317281: Generated new TSK 0
I20260807 03:14:19.202867 8399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:14:19.213160 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 8305
W20260807 03:14:19.214187 8369 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36127
W20260807 03:14:19.214887 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 8305: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:14:19.215058 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 8305
2026-08-07T03:14:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1929 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:14:19 dist-test-slave-tm1g krb5kdc[8407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:14:19 dist-test-slave-tm1g krb5kdc[8407](info): set up 2 sockets
Aug 07 03:14:19 dist-test-slave-tm1g krb5kdc[8407](info): commencing operation
krb5kdc: starting...
W20260807 03:14:20.347855 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:14:20 dist-test-slave-tm1g krb5kdc[8407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:14:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:14:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:14:20.604990 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:42741
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:34711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:42741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:14:20.927969 8421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:20.928217 8421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:20.928371 8421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:14:20.935736 8421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:20.935834 8421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:14:20.935879 8421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:20.935977 8421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:20.936048 8421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:20.945276 8421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:34711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:42741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:42741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:20.946590 8421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:20.947904 8421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:14:20.960054 8427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:20.960130 8426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:20.960151 8429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:20.960830 8421 server_base.cc:1034] running on GCE node
I20260807 03:14:20.961621 8421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:20.963014 8421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:20.964262 8421 hybrid_clock.cc:648] HybridClock initialized: now 1786072460964213 us; error 42 us; skew 500 ppm
I20260807 03:14:20.964612 8421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:20 dist-test-slave-tm1g krb5kdc[8407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072460, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:20.971479 8421 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:14:20.972851 8421 webserver.cc:460] Webserver started at http://127.17.31.62:44227/ using document root <none> and password file <none>
I20260807 03:14:20.973471 8421 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:20.973568 8421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:20.973802 8421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:20.976720 8421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "d132b7dfa9f64682ab3c304296cc0faa"
format_stamp: "Formatted at 2026-08-07 03:14:20 on dist-test-slave-tm1g"
server_key: "b46708d1ad39bb3feea85161e2818f00"
server_key_iv: "55b8ae2c4972ec40a4aefba76ea56c59"
server_key_version: "encryptionkey@0"
I20260807 03:14:20.977409 8421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "d132b7dfa9f64682ab3c304296cc0faa"
format_stamp: "Formatted at 2026-08-07 03:14:20 on dist-test-slave-tm1g"
server_key: "b46708d1ad39bb3feea85161e2818f00"
server_key_iv: "55b8ae2c4972ec40a4aefba76ea56c59"
server_key_version: "encryptionkey@0"
I20260807 03:14:20.982273 8421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260807 03:14:20.985730 8436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:20.987026 8421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:14:20.987150 8421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "d132b7dfa9f64682ab3c304296cc0faa"
format_stamp: "Formatted at 2026-08-07 03:14:20 on dist-test-slave-tm1g"
server_key: "b46708d1ad39bb3feea85161e2818f00"
server_key_iv: "55b8ae2c4972ec40a4aefba76ea56c59"
server_key_version: "encryptionkey@0"
I20260807 03:14:20.987299 8421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:20.992478 8421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:20.996086 8421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:20.996321 8421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:21.014572 8421 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:42741
I20260807 03:14:21.014662 8490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:42741 every 8 connection(s)
I20260807 03:14:21.015902 8421 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:14:21.019331 8491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:21.019897 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 8421
I20260807 03:14:21.020095 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:14:21.020638 17532 external_mini_cluster.cc:1428] Setting key 9e4d22fb87139115c4827b4bc8aba52a
I20260807 03:14:21.028605 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:21.022440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54461 (local address 127.17.31.62:42741)
0807 03:14:21.023248 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:21.023262 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:21.023303 (+ 41us) server_negotiation.cc:407] Connection header received
0807 03:14:21.025494 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:21.025514 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:21.025815 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:21.026135 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:21.027178 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:21.027184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:21.027195 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:14:21.027208 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:21.027262 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:14:21.027283 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:21.027439 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:21.027567 (+ 128us) server_negotiation.cc:299] Negotiation successful
0807 03:14:21.027741 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":365,"threads_started":1}
I20260807 03:14:21.032184 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:14:21.032274 8491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa: Bootstrap starting.
I20260807 03:14:21.035892 8491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:21.037190 8491 log.cc:824] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa: Log is configured to *not* fsync() on all Append() calls
I20260807 03:14:21.039834 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:21.037397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51085 (local address 127.17.31.62:42741)
0807 03:14:21.037591 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:21.037596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:21.038003 (+ 407us) server_negotiation.cc:407] Connection header received
0807 03:14:21.038127 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:21.038136 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:21.038202 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:21.038366 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:21.039211 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:21.039214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:21.039217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 03:14:21.039231 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:21.039265 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 03:14:21.039279 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:21.039362 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:21.039521 (+ 159us) server_negotiation.cc:299] Negotiation successful
0807 03:14:21.039561 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 03:14:21.040558 8491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa: No bootstrap required, opened a new log
I20260807 03:14:21.048120 8491 raft_consensus.cc:348] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d132b7dfa9f64682ab3c304296cc0faa" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42741 } }
I20260807 03:14:21.048483 8491 raft_consensus.cc:374] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 03:14:21.048595 8491 raft_consensus.cc:729] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d132b7dfa9f64682ab3c304296cc0faa, State: Initialized, Role: FOLLOWER
I20260807 03:14:21.049137 8491 consensus_queue.cc:260] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d132b7dfa9f64682ab3c304296cc0faa" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42741 } }
I20260807 03:14:21.049391 8491 raft_consensus.cc:388] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 03:14:21.049546 8491 raft_consensus.cc:482] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 03:14:21.049741 8491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:21.052134 8491 raft_consensus.cc:504] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d132b7dfa9f64682ab3c304296cc0faa" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42741 } }
I20260807 03:14:21.052711 8491 leader_election.cc:302] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d132b7dfa9f64682ab3c304296cc0faa; no voters:
I20260807 03:14:21.053205 8491 leader_election.cc:288] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 03:14:21.053354 8502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:14:21.054054 8502 raft_consensus.cc:686] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [term 1 LEADER]: Becoming Leader. State: Replica: d132b7dfa9f64682ab3c304296cc0faa, State: Running, Role: LEADER
I20260807 03:14:21.054605 8502 consensus_queue.cc:237] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d132b7dfa9f64682ab3c304296cc0faa" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42741 } }
I20260807 03:14:21.055287 8491 sys_catalog.cc:564] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:14:21.057446 8504 sys_catalog.cc:455] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d132b7dfa9f64682ab3c304296cc0faa. Latest consensus state: current_term: 1 leader_uuid: "d132b7dfa9f64682ab3c304296cc0faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d132b7dfa9f64682ab3c304296cc0faa" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42741 } } }
I20260807 03:14:21.057840 8504 sys_catalog.cc:458] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:21.060108 8507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:14:21.060081 8503 sys_catalog.cc:455] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d132b7dfa9f64682ab3c304296cc0faa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d132b7dfa9f64682ab3c304296cc0faa" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 42741 } } }
I20260807 03:14:21.060441 8503 sys_catalog.cc:458] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:21.065157 8507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:14:21.073714 8507 catalog_manager.cc:1347] Generated new cluster ID: 7347d52208264b9c9d560fc51d53f193
I20260807 03:14:21.073830 8507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:14:21.083597 8507 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:14:21.083977 8507 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:14:21.090291 8507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d132b7dfa9f64682ab3c304296cc0faa: Generated new TSK 0
I20260807 03:14:21.091063 8507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:14:21.097509 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 8421
W20260807 03:14:21.098922 8486 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54461
W20260807 03:14:21.099591 17532 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 8421: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260807 03:14:21.099745 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 8421
2026-08-07T03:14:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1881 ms)
[----------] 18 tests from AuthTokenIssuingTest (35330 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:14:21 dist-test-slave-tm1g krb5kdc[8523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:14:21 dist-test-slave-tm1g krb5kdc[8523](info): set up 2 sockets
Aug 07 03:14:21 dist-test-slave-tm1g krb5kdc[8523](info): commencing operation
krb5kdc: starting...
W20260807 03:14:22.232887 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:14:22 dist-test-slave-tm1g krb5kdc[8523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:14:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:14:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:14:22.505332 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:46781
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:46781,127.17.31.61:36549,127.17.31.60:43723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:14:22.837070 8536 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 03:14:22.837378 8536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:22.837464 8536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:22.837549 8536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:14:22.844609 8536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:22.844698 8536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:14:22.844755 8536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:22.844805 8536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:22.844861 8536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:22.854194 8536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:46781,127.17.31.61:36549,127.17.31.60:43723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:46781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:22.855362 8536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:22.856624 8536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:14:22.867983 8541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:22.867873 8542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:22.868006 8544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:22.868929 8536 server_base.cc:1034] running on GCE node
I20260807 03:14:22.869700 8536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:22.871111 8536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:22.872361 8536 hybrid_clock.cc:648] HybridClock initialized: now 1786072462872317 us; error 50 us; skew 500 ppm
I20260807 03:14:22.872685 8536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:22 dist-test-slave-tm1g krb5kdc[8523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072462, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:22.875896 8536 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:14:22.877264 8536 webserver.cc:460] Webserver started at http://127.17.31.62:39505/ using document root <none> and password file <none>
I20260807 03:14:22.877962 8536 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:22.878105 8536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:22.878508 8536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:22.881358 8536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "605ac0ec3e87491d93c2c7e3530ed2e1"
format_stamp: "Formatted at 2026-08-07 03:14:22 on dist-test-slave-tm1g"
server_key: "0ff1087c6fd6e5dc53208691b7dd9772"
server_key_iv: "b7b3c7f514631e5980f4522c8acd5d2c"
server_key_version: "encryptionkey@0"
I20260807 03:14:22.882159 8536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "605ac0ec3e87491d93c2c7e3530ed2e1"
format_stamp: "Formatted at 2026-08-07 03:14:22 on dist-test-slave-tm1g"
server_key: "0ff1087c6fd6e5dc53208691b7dd9772"
server_key_iv: "b7b3c7f514631e5980f4522c8acd5d2c"
server_key_version: "encryptionkey@0"
I20260807 03:14:22.887120 8536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 03:14:22.890503 8551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:22.891748 8536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:14:22.891952 8536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "605ac0ec3e87491d93c2c7e3530ed2e1"
format_stamp: "Formatted at 2026-08-07 03:14:22 on dist-test-slave-tm1g"
server_key: "0ff1087c6fd6e5dc53208691b7dd9772"
server_key_iv: "b7b3c7f514631e5980f4522c8acd5d2c"
server_key_version: "encryptionkey@0"
I20260807 03:14:22.892170 8536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:22.914515 8536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:22.918859 8536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:22.919152 8536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:22.937742 8536 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:46781
I20260807 03:14:22.937765 8604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:46781 every 8 connection(s)
I20260807 03:14:22.939117 8536 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:14:22.940811 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 8536
I20260807 03:14:22.941030 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:14:22.941674 17532 external_mini_cluster.cc:1428] Setting key 25db225645fccff6790aacbb9df7bd58
I20260807 03:14:22.942922 8605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:22.949146 8605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:14:22 dist-test-slave-tm1g krb5kdc[8523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072462, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.31.61@KRBTEST.COM; defaulting to no policy
I20260807 03:14:22.974951 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:22.953308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.17.31.62:46781)
0807 03:14:22.953943 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:22.953948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:22.953991 (+ 43us) server_negotiation.cc:407] Connection header received
0807 03:14:22.956422 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:22.956429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:22.956651 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:22.956810 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:22.958456 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:22.959659 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:22.962008 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:22.962612 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:22.962742 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:22.966766 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:22.966780 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:22.966799 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:22.966851 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:22.970827 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:22.971514 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:22.971524 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:22.971529 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:22.971743 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:22.972188 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:22.972194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:22.972197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:22.972914 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:22.973123 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:22.973677 (+ 554us) server_negotiation.cc:299] Negotiation successful
0807 03:14:22.973995 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":470,"thread_start_us":338,"threads_started":1}
I20260807 03:14:22.975000 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:22.952305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:46781 (local address 127.0.0.1:47348)
0807 03:14:22.953151 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:22.953211 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:22.956023 (+ 2812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:22.956895 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:22.956904 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:22.957728 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:22.958227 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:22.958252 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:22.959829 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:22.959837 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:22.961864 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:22.961883 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:22.962871 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:22.962882 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:22.963125 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:22.963803 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:22.963836 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:22.966520 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:22.970964 (+ 4444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:22.970974 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:22.970982 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:22.971346 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:22.971861 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:22.971864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:22.971866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:22.972043 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:22.973051 (+ 1008us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:22.973056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:22.973235 (+ 179us) client_negotiation.cc:769] Sending connection context
0807 03:14:22.973497 (+ 262us) client_negotiation.cc:240] Negotiation successful
0807 03:14:22.973855 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":320,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.61@KRBTEST.COM" created.
I20260807 03:14:22.977896 8605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:14:22.978972 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:22.978583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:36549 (local address 127.0.0.1:49462)
0807 03:14:22.978762 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:22.978874 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260807 03:14:22.979274 8554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.31.61:36549: connect: Connection refused (error 111)
I20260807 03:14:22.980821 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:22.980499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:36549 (local address 127.0.0.1:49468)
0807 03:14:22.980651 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:22.980710 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 03:14:22.981026 8605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.61:36549: Network error: Client connection negotiation failed: client connection to 127.17.31.61:36549: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.61.keytab.
Entry for principal kudu/127.17.31.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.61.keytab.
I20260807 03:14:23.010001 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.61:36549
--webserver_interface=127.17.31.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.61.keytab
--principal=kudu/127.17.31.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:46781,127.17.31.61:36549,127.17.31.60:43723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:14:23.030390 8605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } attempt: 1
I20260807 03:14:23.031833 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.031217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:36549 (local address 127.0.0.1:49472)
0807 03:14:23.031495 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.031623 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260807 03:14:23.032953 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.032493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:36549 (local address 127.0.0.1:49478)
0807 03:14:23.032681 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.032779 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260807 03:14:23.033198 8605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.61:36549: Network error: Client connection negotiation failed: client connection to 127.17.31.61:36549: connect: Connection refused (error 111)
I20260807 03:14:23.101492 8605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } attempt: 2
I20260807 03:14:23.102810 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.102226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:36549 (local address 127.0.0.1:49486)
0807 03:14:23.102521 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.102636 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260807 03:14:23.103924 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.103494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:36549 (local address 127.0.0.1:49496)
0807 03:14:23.103667 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.103753 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260807 03:14:23.104132 8605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.61:36549: Network error: Client connection negotiation failed: client connection to 127.17.31.61:36549: connect: Connection refused (error 111)
I20260807 03:14:23.195426 8605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } attempt: 3
I20260807 03:14:23.196786 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.196174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:36549 (local address 127.0.0.1:49512)
0807 03:14:23.196445 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.196579 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260807 03:14:23.197867 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.197519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:36549 (local address 127.0.0.1:49518)
0807 03:14:23.197657 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.197726 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 03:14:23.198169 8605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.61:36549: Network error: Client connection negotiation failed: client connection to 127.17.31.61:36549: connect: Connection refused (error 111)
I20260807 03:14:23.341398 8605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } attempt: 4
I20260807 03:14:23.342645 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.342094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:36549 (local address 127.0.0.1:49528)
0807 03:14:23.342407 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.342509 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260807 03:14:23.343703 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.343333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:36549 (local address 127.0.0.1:49532)
0807 03:14:23.343473 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.343574 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:36549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 03:14:23.343946 8605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.61:36549: Network error: Client connection negotiation failed: client connection to 127.17.31.61:36549: connect: Connection refused (error 111)
W20260807 03:14:23.344244 8613 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 03:14:23.344576 8613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:23.344671 8613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:23.344800 8613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:14:23.352084 8613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:23.352165 8613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:14:23.352306 8613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:23.352380 8613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:23.352484 8613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:23.362985 8613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:46781,127.17.31.61:36549,127.17.31.60:43723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.61.keytab
--principal=kudu/127.17.31.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.61:36549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.31.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:23.364228 8613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:23.365495 8613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:14:23.376709 8618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:23.376753 8621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:23.376668 8619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:23.377480 8613 server_base.cc:1034] running on GCE node
I20260807 03:14:23.378245 8613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:23.379505 8613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:23.380741 8613 hybrid_clock.cc:648] HybridClock initialized: now 1786072463380686 us; error 35 us; skew 500 ppm
I20260807 03:14:23.381057 8613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:23 dist-test-slave-tm1g krb5kdc[8523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072463, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:23.384123 8613 init.cc:377] Logged in from keytab as kudu/127.17.31.61@KRBTEST.COM (short username kudu)
I20260807 03:14:23.385426 8613 webserver.cc:460] Webserver started at http://127.17.31.61:45447/ using document root <none> and password file <none>
I20260807 03:14:23.386086 8613 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:23.386214 8613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:23.386571 8613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:23.389381 8613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/data/instance:
uuid: "a6fa76da8bcf4c30b1b55dcaf617911a"
format_stamp: "Formatted at 2026-08-07 03:14:23 on dist-test-slave-tm1g"
server_key: "c7bd4bfbd993088721c26b160ff20ecf"
server_key_iv: "64372ceb3107d38bacce3092af9eff3e"
server_key_version: "encryptionkey@0"
I20260807 03:14:23.390228 8613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/wal/instance:
uuid: "a6fa76da8bcf4c30b1b55dcaf617911a"
format_stamp: "Formatted at 2026-08-07 03:14:23 on dist-test-slave-tm1g"
server_key: "c7bd4bfbd993088721c26b160ff20ecf"
server_key_iv: "64372ceb3107d38bacce3092af9eff3e"
server_key_version: "encryptionkey@0"
I20260807 03:14:23.395105 8613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 03:14:23.398433 8629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:23.399657 8613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 03:14:23.399827 8613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/wal
uuid: "a6fa76da8bcf4c30b1b55dcaf617911a"
format_stamp: "Formatted at 2026-08-07 03:14:23 on dist-test-slave-tm1g"
server_key: "c7bd4bfbd993088721c26b160ff20ecf"
server_key_iv: "64372ceb3107d38bacce3092af9eff3e"
server_key_version: "encryptionkey@0"
I20260807 03:14:23.400031 8613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:23.417160 8613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:23.420758 8613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:23.421020 8613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:23.439517 8613 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.61:36549
I20260807 03:14:23.439558 8681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.61:36549 every 8 connection(s)
I20260807 03:14:23.440758 8613 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/data/info.pb
I20260807 03:14:23.444311 8682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:23.445062 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 8613
I20260807 03:14:23.445271 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-1/wal/instance
I20260807 03:14:23.445837 17532 external_mini_cluster.cc:1428] Setting key ed9761d1f3b922ad0be8413c25d824e5
I20260807 03:14:23.454030 8682 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:14:23 dist-test-slave-tm1g krb5kdc[8523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072463, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.61@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:14:23.476526 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.457263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47352 (local address 127.17.31.62:46781)
0807 03:14:23.457489 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:23.457496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:23.458760 (+ 1264us) server_negotiation.cc:407] Connection header received
0807 03:14:23.460585 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:23.460594 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:23.460686 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:23.460802 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:23.462188 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.463394 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.464842 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.465649 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.465792 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:23.469373 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:23.469382 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:23.469389 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:23.469435 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:23.473347 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:23.474019 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:23.474028 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:23.474033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:23.474196 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:23.474648 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:23.474656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:23.474662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:23.475082 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:23.475242 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:23.475877 (+ 635us) server_negotiation.cc:299] Negotiation successful
0807 03:14:23.476133 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 03:14:23.477104 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.457724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:46781 (local address 127.0.0.1:47352)
0807 03:14:23.458577 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.458635 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:23.460316 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:23.460984 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:23.460991 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:23.461476 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:23.462016 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:23.462033 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.463551 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.463555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:23.464689 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:23.464701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.465854 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.465859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:23.466061 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:23.466782 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:23.466804 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:23.469151 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:23.473488 (+ 4337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:23.473494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:23.473505 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:23.473895 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:23.474349 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:23.474358 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:23.474362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:23.474533 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:23.475199 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:23.475205 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:23.475502 (+ 297us) client_negotiation.cc:769] Sending connection context
0807 03:14:23.475708 (+ 206us) client_negotiation.cc:240] Negotiation successful
0807 03:14:23.476048 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":387,"threads_started":1}
WARNING: no policy specified for kudu/127.17.31.60@KRBTEST.COM; defaulting to no policy
I20260807 03:14:23.478827 8682 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.60@KRBTEST.COM" created.
Aug 07 03:14:23 dist-test-slave-tm1g krb5kdc[8523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072463, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.61@KRBTEST.COM for kudu/127.17.31.61@KRBTEST.COM
I20260807 03:14:23.494858 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.479777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49536 (local address 127.17.31.61:36549)
0807 03:14:23.480515 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:23.480523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:23.480556 (+ 33us) server_negotiation.cc:407] Connection header received
0807 03:14:23.480611 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:23.480618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:23.480842 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:23.480979 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:23.481669 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.483431 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.484539 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.485466 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.485618 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:23.488665 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:23.488681 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:23.488698 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:23.488750 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:23.492005 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:23.492811 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:23.492817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:23.492819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:23.492930 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:23.493326 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:23.493329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:23.493331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:23.493647 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:23.493790 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:23.494231 (+ 441us) server_negotiation.cc:299] Negotiation successful
0807 03:14:23.494572 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":542,"spinlock_wait_cycles":18176,"thread_start_us":405,"threads_started":1}
I20260807 03:14:23.494961 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.479585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:36549 (local address 127.0.0.1:49536)
0807 03:14:23.479758 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.479773 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:23.479852 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:23.481012 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:23.481015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:23.481369 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:23.481502 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:23.481510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.483582 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.483585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:23.484406 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:23.484417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.485617 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.485622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:23.485756 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:23.486297 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:23.486374 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:23.488438 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:23.492139 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:23.492146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:23.492153 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:23.492695 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:23.493051 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:23.493059 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:23.493065 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:23.493225 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:23.493766 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:23.493773 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:23.493948 (+ 175us) client_negotiation.cc:769] Sending connection context
0807 03:14:23.494116 (+ 168us) client_negotiation.cc:240] Negotiation successful
0807 03:14:23.494460 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":2}
I20260807 03:14:23.498077 8682 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:14:23.499356 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.498788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44776)
0807 03:14:23.498990 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.499166 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 03:14:23.499665 8631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
I20260807 03:14:23.501333 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.500893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44790)
0807 03:14:23.501077 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.501189 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 03:14:23.501564 8682 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:43723: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.60.keytab.
Entry for principal kudu/127.17.31.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.60.keytab.
I20260807 03:14:23.515245 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.60:43723
--webserver_interface=127.17.31.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.60.keytab
--principal=kudu/127.17.31.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:39099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:46781,127.17.31.61:36549,127.17.31.60:43723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:14:23.550904 8682 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } attempt: 1
I20260807 03:14:23.552139 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.551587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44792)
0807 03:14:23.551817 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.551950 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 03:14:23.553130 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.552744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44800)
0807 03:14:23.552904 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.552989 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 03:14:23.553347 8682 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:43723: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
I20260807 03:14:23.621582 8682 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } attempt: 2
I20260807 03:14:23.622825 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.622380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44816)
0807 03:14:23.622575 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.622665 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 03:14:23.623821 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.623519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44824)
0807 03:14:23.623645 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.623711 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 03:14:23.624063 8682 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:43723: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
I20260807 03:14:23.643175 8605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } attempt: 5
Aug 07 03:14:23 dist-test-slave-tm1g krb5kdc[8523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072462, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for kudu/127.17.31.61@KRBTEST.COM
I20260807 03:14:23.656863 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.643981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.17.31.61:36549)
0807 03:14:23.644180 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:23.644186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:23.644210 (+ 24us) server_negotiation.cc:407] Connection header received
0807 03:14:23.644335 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:23.644340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:23.644420 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:23.644526 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:23.645327 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.646445 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.647578 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.648262 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.648405 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:23.651786 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:23.651791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:23.651794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:23.651824 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:23.654362 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:23.655152 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:23.655156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:23.655158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:23.655272 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:23.655700 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:23.655703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:23.655705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:23.655967 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:23.656095 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:23.656514 (+ 419us) server_negotiation.cc:299] Negotiation successful
0807 03:14:23.656664 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 03:14:23.657160 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.643907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:36549 (local address 127.0.0.1:49550)
0807 03:14:23.644094 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.644118 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:23.644219 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:23.644572 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:23.644576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:23.644961 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:23.645144 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:23.645153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.646564 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.646568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:23.647442 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:23.647453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.648409 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.648430 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:23.648569 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:23.649217 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:23.649234 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:23.651614 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:23.654493 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:23.654498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:23.654503 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:23.655047 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:23.655386 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:23.655393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:23.655398 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:23.655590 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:23.656099 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:23.656104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:23.656249 (+ 145us) client_negotiation.cc:769] Sending connection context
0807 03:14:23.656431 (+ 182us) client_negotiation.cc:240] Negotiation successful
0807 03:14:23.656693 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 03:14:23.658174 8605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:14:23.659374 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.659028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44838)
0807 03:14:23.659178 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.659246 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260807 03:14:23.660346 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.659997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44840)
0807 03:14:23.660159 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.660221 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 03:14:23.660565 8605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:43723: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
I20260807 03:14:23.711779 8605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } attempt: 1
I20260807 03:14:23.712983 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.712516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44854)
0807 03:14:23.712715 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.712806 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260807 03:14:23.714006 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.713679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44870)
0807 03:14:23.713826 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.713893 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 03:14:23.714210 8605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:43723: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
I20260807 03:14:23.715258 8682 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } attempt: 3
I20260807 03:14:23.716287 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.715838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44876)
0807 03:14:23.716029 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.716108 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 03:14:23.717283 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.716869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44890)
0807 03:14:23.717075 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.717161 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260807 03:14:23.717495 8682 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:43723: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
I20260807 03:14:23.782411 8605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } attempt: 2
I20260807 03:14:23.783555 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.783072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44904)
0807 03:14:23.783293 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.783389 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260807 03:14:23.784555 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.784235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44912)
0807 03:14:23.784400 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.784462 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 03:14:23.784756 8605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:43723: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
W20260807 03:14:23.860140 8689 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 03:14:23.860493 8689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:23.860630 8689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:23.860759 8689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260807 03:14:23.860674 8682 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } attempt: 4
I20260807 03:14:23.861806 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.861334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44922)
0807 03:14:23.861563 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.861652 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260807 03:14:23.862758 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.862455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44928)
0807 03:14:23.862595 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.862666 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 03:14:23.862975 8682 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:43723: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
W20260807 03:14:23.868072 8689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:23.868177 8689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:14:23.868295 8689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:23.868386 8689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:23.868490 8689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:23.878562 8689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:39099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:46781,127.17.31.61:36549,127.17.31.60:43723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.60.keytab
--principal=kudu/127.17.31.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.60:43723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.31.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:23.879765 8689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:23.881038 8689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 03:14:23.890957 8605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } attempt: 3
I20260807 03:14:23.892246 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.891681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44938)
0807 03:14:23.891869 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.891970 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260807 03:14:23.892385 8697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:23.893025 8694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:23.893400 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.892899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44942)
0807 03:14:23.893172 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.893274 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
W20260807 03:14:23.893460 8695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:23.893656 8605 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:43723: Network error: Client connection negotiation failed: client connection to 127.17.31.60:43723: connect: Connection refused (error 111)
I20260807 03:14:23.893908 8689 server_base.cc:1034] running on GCE node
I20260807 03:14:23.894701 8689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:23.896075 8689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:23.897307 8689 hybrid_clock.cc:648] HybridClock initialized: now 1786072463897229 us; error 51 us; skew 500 ppm
I20260807 03:14:23.897619 8689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:23 dist-test-slave-tm1g krb5kdc[8523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072463, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:23.901052 8689 init.cc:377] Logged in from keytab as kudu/127.17.31.60@KRBTEST.COM (short username kudu)
I20260807 03:14:23.902451 8689 webserver.cc:460] Webserver started at http://127.17.31.60:35599/ using document root <none> and password file <none>
I20260807 03:14:23.903169 8689 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:23.903287 8689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:23.903657 8689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:23.906672 8689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/data/instance:
uuid: "460527176d924a0faf42ee2f867546d9"
format_stamp: "Formatted at 2026-08-07 03:14:23 on dist-test-slave-tm1g"
server_key: "b8a1cf18a76ba31f71097e97775cb059"
server_key_iv: "ad640771d465bd20d2677cd04504bbff"
server_key_version: "encryptionkey@0"
I20260807 03:14:23.907440 8689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/wal/instance:
uuid: "460527176d924a0faf42ee2f867546d9"
format_stamp: "Formatted at 2026-08-07 03:14:23 on dist-test-slave-tm1g"
server_key: "b8a1cf18a76ba31f71097e97775cb059"
server_key_iv: "ad640771d465bd20d2677cd04504bbff"
server_key_version: "encryptionkey@0"
I20260807 03:14:23.912333 8689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260807 03:14:23.915661 8704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:23.916854 8689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 03:14:23.917057 8689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/wal
uuid: "460527176d924a0faf42ee2f867546d9"
format_stamp: "Formatted at 2026-08-07 03:14:23 on dist-test-slave-tm1g"
server_key: "b8a1cf18a76ba31f71097e97775cb059"
server_key_iv: "ad640771d465bd20d2677cd04504bbff"
server_key_version: "encryptionkey@0"
I20260807 03:14:23.917322 8689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:23.931686 8689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:23.935386 8689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:23.935653 8689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:23.954908 8689 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.60:43723
I20260807 03:14:23.954941 8758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.60:43723 every 8 connection(s)
I20260807 03:14:23.956115 8689 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/data/info.pb
I20260807 03:14:23.959744 8759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:23.960305 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 8689
I20260807 03:14:23.960489 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786072106928531-17532-0/minicluster-data/master-2/wal/instance
I20260807 03:14:23.960937 17532 external_mini_cluster.cc:1428] Setting key 928be5328d4189355b2354bd5d769a73
I20260807 03:14:23.969255 8759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:14:23 dist-test-slave-tm1g krb5kdc[8523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.60@KRBTEST.COM
Aug 07 03:14:23 dist-test-slave-tm1g krb5kdc[8523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072463, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.60@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:14:23.984071 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.962729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.17.31.60:43723)
0807 03:14:23.963531 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:23.963544 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:23.963597 (+ 53us) server_negotiation.cc:407] Connection header received
0807 03:14:23.966233 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:23.966259 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:23.966761 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:23.967192 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:23.968400 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.970420 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.971795 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.972655 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.972854 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:23.976752 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:23.976759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:23.976768 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:23.976800 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:23.980121 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:23.980898 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:23.980904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:23.980909 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:23.981060 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:23.981495 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:23.981499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:23.981501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:23.981940 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:23.982239 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:23.982650 (+ 411us) server_negotiation.cc:299] Negotiation successful
0807 03:14:23.982921 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":338,"threads_started":1}
I20260807 03:14:23.986389 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.972000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.17.31.62:46781)
0807 03:14:23.972213 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:23.972218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:23.973047 (+ 829us) server_negotiation.cc:407] Connection header received
0807 03:14:23.973317 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:23.973322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:23.973395 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:23.973494 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:23.974569 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.975269 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.976638 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.977086 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.977232 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:23.980579 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:23.980586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:23.980591 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:23.980634 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:23.983565 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:23.984391 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:23.984397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:23.984401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:23.984545 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:23.984928 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:23.984931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:23.984933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:23.985177 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:23.985352 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:23.985784 (+ 432us) server_negotiation.cc:299] Negotiation successful
0807 03:14:23.986000 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260807 03:14:23.986436 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.972091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:46781 (local address 127.0.0.1:47364)
0807 03:14:23.972916 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.972963 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:23.973166 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:23.973552 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:23.973558 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:23.974134 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:23.974430 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:23.974439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.975417 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.975422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:23.976494 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:23.976505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.977292 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.977300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:23.977465 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:23.978061 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:23.978080 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:23.980397 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:23.983694 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:23.983705 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:23.983711 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:23.984276 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:23.984652 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:23.984657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:23.984660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:23.984827 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:23.985317 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:23.985326 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:23.985507 (+ 181us) client_negotiation.cc:769] Sending connection context
0807 03:14:23.985692 (+ 185us) client_negotiation.cc:240] Negotiation successful
0807 03:14:23.985977 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":454,"threads_started":1}
I20260807 03:14:23.988049 8759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:14:23 dist-test-slave-tm1g krb5kdc[8523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072463, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.60@KRBTEST.COM for kudu/127.17.31.61@KRBTEST.COM
Aug 07 03:14:23 dist-test-slave-tm1g krb5kdc[8523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:14:24.002009 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.988795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:36549 (local address 127.0.0.1:49562)
0807 03:14:23.988943 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:23.988960 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:23.989050 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:23.989321 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:23.989325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:23.989743 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:23.989911 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:23.989920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.991234 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.991237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:23.992418 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:23.992435 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.993425 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.993430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:23.993586 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:23.994074 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:23.994090 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:23.996042 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:23.999728 (+ 3686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:23.999731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:23.999733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:24.000067 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:24.000457 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:24.000465 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:24.000472 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:24.000654 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:24.001234 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:24.001242 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:24.001390 (+ 148us) client_negotiation.cc:769] Sending connection context
0807 03:14:24.001495 (+ 105us) client_negotiation.cc:240] Negotiation successful
0807 03:14:24.001698 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260807 03:14:24.002177 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.988823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49562 (local address 127.17.31.61:36549)
0807 03:14:23.989043 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:23.989049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:23.989071 (+ 22us) server_negotiation.cc:407] Connection header received
0807 03:14:23.989147 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:23.989151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:23.989205 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:23.989273 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:23.990038 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.991108 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.992571 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.993246 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.993416 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:23.996210 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:23.996214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:23.996218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:23.996246 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:23.999603 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.000179 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.000186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.000189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.000348 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.000773 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.000777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.000781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.001112 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:24.001225 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.001614 (+ 389us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.001844 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260807 03:14:24.002980 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:23.987191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47368 (local address 127.17.31.62:46781)
0807 03:14:23.987361 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:23.987368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:23.987385 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:14:23.987449 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:23.987456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:23.987528 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:23.987614 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:23.989038 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.990063 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.991705 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:23.992367 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:23.992514 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:23.996898 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:23.996903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:23.996906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:23.996955 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:24.000084 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.000774 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.000780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.000783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.000923 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.001303 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.001306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.001308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.001616 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:24.001881 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.002464 (+ 583us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.002679 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:14:24.003335 8759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:14:24 dist-test-slave-tm1g krb5kdc[8523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072463, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.60@KRBTEST.COM for kudu/127.17.31.60@KRBTEST.COM
Aug 07 03:14:24 dist-test-slave-tm1g krb5kdc[8523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.61@KRBTEST.COM
I20260807 03:14:24.015949 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.004000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44948)
0807 03:14:24.004140 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:24.004157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:24.004245 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:24.004573 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:24.004578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:24.004976 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:24.005153 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:24.005165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.006055 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.006063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:24.006966 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:24.006977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.007933 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.007942 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:24.008206 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.008757 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:24.008773 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:24.010587 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:24.013589 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:24.013596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:24.013600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:24.013931 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:24.014262 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:24.014269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:24.014274 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:24.014448 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:24.014942 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:24.014946 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:24.015076 (+ 130us) client_negotiation.cc:769] Sending connection context
0807 03:14:24.015197 (+ 121us) client_negotiation.cc:240] Negotiation successful
0807 03:14:24.015549 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260807 03:14:24.016160 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.004002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.17.31.60:43723)
0807 03:14:24.004165 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.004170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.004206 (+ 36us) server_negotiation.cc:407] Connection header received
0807 03:14:24.004375 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.004383 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.004449 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.004522 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.005271 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.005939 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.007103 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.007598 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.007738 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.010736 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:24.010742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:24.010746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:24.010780 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:24.013476 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.014038 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.014045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.014048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.014163 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.014556 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.014560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.014563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.014813 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:24.014973 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.015285 (+ 312us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.015712 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":210,"server-negotiator.queue_time_us":45}
I20260807 03:14:24.019209 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.005008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49572 (local address 127.17.31.61:36549)
0807 03:14:24.005164 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.005170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.005187 (+ 17us) server_negotiation.cc:407] Connection header received
0807 03:14:24.005321 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.005327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.005392 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.005485 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.006657 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.007336 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.009012 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.009441 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.009589 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.013903 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:24.013907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:24.013910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:24.013938 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:24.016777 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.017509 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.017514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.017517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.017645 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.018061 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.018064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.018066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.018299 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:24.018535 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.018810 (+ 275us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.018961 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 03:14:24.021831 8759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9: Bootstrap starting.
I20260807 03:14:24.023934 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:14:24.025374 8759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:24.026523 8759 log.cc:824] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9: Log is configured to *not* fsync() on all Append() calls
I20260807 03:14:24.029182 8759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9: No bootstrap required, opened a new log
I20260807 03:14:24.026585 8578 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47368
I20260807 03:14:24.036227 8759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } }
I20260807 03:14:24.036554 8759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 460527176d924a0faf42ee2f867546d9, State: Initialized, Role: FOLLOWER
I20260807 03:14:24.036957 8759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } }
I20260807 03:14:24.037835 8765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } } }
I20260807 03:14:24.038237 8765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:14:24.038937 8759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 03:14:24.048600 8776 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:14:24.048724 8776 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:14:24.046066 8689 master_runner.cc:186] Error getting master registration for 127.17.31.62:46781: OK, instance_id { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" instance_seqno: 1786072462932478 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 03:14:24.057366 8689 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 03:14:24.070901 8605 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } attempt: 4
Aug 07 03:14:24 dist-test-slave-tm1g krb5kdc[8523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072462, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for kudu/127.17.31.60@KRBTEST.COM
I20260807 03:14:24.087888 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.071682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44964)
0807 03:14:24.071900 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:24.071923 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:24.072049 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:24.072440 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:24.072444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:24.072998 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:24.073218 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:24.073229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.074757 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.074762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:24.076252 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:24.076270 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.077328 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.077336 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:24.077551 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.078286 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:24.078355 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:24.081010 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:24.085096 (+ 4086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:24.085101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:24.085106 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:24.085623 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:24.086036 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:24.086040 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:24.086046 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:24.086209 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:24.086848 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:24.086856 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:24.087028 (+ 172us) client_negotiation.cc:769] Sending connection context
0807 03:14:24.087179 (+ 151us) client_negotiation.cc:240] Negotiation successful
0807 03:14:24.087475 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 03:14:24.087903 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.071870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.17.31.60:43723)
0807 03:14:24.072094 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.072100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.072127 (+ 27us) server_negotiation.cc:407] Connection header received
0807 03:14:24.072198 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.072204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.072293 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.072404 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.073383 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.074606 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.076404 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.077172 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.077297 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.081209 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:24.081216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:24.081221 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:24.081259 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:24.084967 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.085742 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.085748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.085753 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.085903 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.086344 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.086353 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.086356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.086700 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:24.086894 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.087312 (+ 418us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.087559 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 03:14:24.094089 8605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1: Bootstrap starting.
I20260807 03:14:24.098466 8605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:24.099872 8605 log.cc:824] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1: Log is configured to *not* fsync() on all Append() calls
I20260807 03:14:24.103318 8605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1: No bootstrap required, opened a new log
I20260807 03:14:24.110999 8605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } }
I20260807 03:14:24.111403 8605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605ac0ec3e87491d93c2c7e3530ed2e1, State: Initialized, Role: FOLLOWER
I20260807 03:14:24.111913 8605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } }
I20260807 03:14:24.112944 8779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } } }
I20260807 03:14:24.113327 8779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:14:24.114177 8605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:14:24.114377 8578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:14:24.114588 8578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:24.117796 8578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } }
I20260807 03:14:24.120649 8578 leader_election.cc:288] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [CANDIDATE]: Term 1 election: Requested vote from peers a6fa76da8bcf4c30b1b55dcaf617911a (127.17.31.61:36549), 460527176d924a0faf42ee2f867546d9 (127.17.31.60:43723)
I20260807 03:14:24.120573 8733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "460527176d924a0faf42ee2f867546d9"
I20260807 03:14:24.121124 8733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:24.125237 8733 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 605ac0ec3e87491d93c2c7e3530ed2e1 in term 1.
I20260807 03:14:24.125988 8553 leader_election.cc:302] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 460527176d924a0faf42ee2f867546d9, 605ac0ec3e87491d93c2c7e3530ed2e1; no voters:
I20260807 03:14:24.126950 8779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [term 1 FOLLOWER]: Leader election won for term 1
W20260807 03:14:24.131770 8790 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:14:24.132015 8790 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:14:24.138573 8779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [term 1 LEADER]: Becoming Leader. State: Replica: 605ac0ec3e87491d93c2c7e3530ed2e1, State: Running, Role: LEADER
I20260807 03:14:24.139323 8779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } }
I20260807 03:14:24.145364 8794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 605ac0ec3e87491d93c2c7e3530ed2e1. Latest consensus state: current_term: 1 leader_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } } }
I20260807 03:14:24.145886 8794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:24.154898 8802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:14:24.157266 8802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:14:24.162160 8682 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } attempt: 5
I20260807 03:14:24.150862 8536 master_runner.cc:186] Error getting master registration for 127.17.31.61:36549: OK, instance_id { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" instance_seqno: 1786072463434226 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 03:14:24.171160 8536 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260807 03:14:24.171806 8557 tablet.cc:2369] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:14:24.186030 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.130986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44968 (local address 127.17.31.60:43723)
0807 03:14:24.131157 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.131162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.144874 (+ 13712us) server_negotiation.cc:407] Connection header received
0807 03:14:24.145094 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.145101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.145186 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.145282 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.148238 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.149244 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.152797 (+ 3553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.153478 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.153581 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.166909 (+ 13328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:24.166921 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:24.166926 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:24.166969 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:24.170884 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.176987 (+ 6103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.176993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.176998 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.177189 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.182240 (+ 5051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.182245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.182249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.182672 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:24.182935 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.185332 (+ 2397us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.185626 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 03:14:24.187014 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.131935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47384 (local address 127.17.31.62:46781)
0807 03:14:24.132095 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.132104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.139430 (+ 7326us) server_negotiation.cc:407] Connection header received
0807 03:14:24.140382 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.140388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.140471 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.140567 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.145356 (+ 4789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.146613 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.151616 (+ 5003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.152448 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.152556 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.171844 (+ 19288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:24.171852 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:24.171856 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:24.171948 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:24.176128 (+ 4180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.177692 (+ 1564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.177698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.177702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.177866 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.184074 (+ 6208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.184080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.184083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.184456 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:24.184726 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.186383 (+ 1657us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.186651 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260807 03:14:24.187795 8710 tablet.cc:2369] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 07 03:14:24 dist-test-slave-tm1g krb5kdc[8523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072463, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.61@KRBTEST.COM for kudu/127.17.31.60@KRBTEST.COM
I20260807 03:14:24.201752 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.130468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49578 (local address 127.17.31.61:36549)
0807 03:14:24.130708 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.130716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.140175 (+ 9459us) server_negotiation.cc:407] Connection header received
0807 03:14:24.140448 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.140454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.140537 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.140606 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.154566 (+ 13960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.155682 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.162434 (+ 6752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.163120 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.163224 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.178411 (+ 15187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:24.178418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:24.178423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:24.178465 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:24.182058 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.187747 (+ 5689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.187754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.187760 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.187966 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.199934 (+ 11968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.199940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.199943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.200300 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:24.200540 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.201126 (+ 586us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.201375 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260807 03:14:24.204497 8733 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9 [term 1 FOLLOWER]: Refusing update from remote peer 605ac0ec3e87491d93c2c7e3530ed2e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:14:24.205861 8794 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:14:24.209676 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.162742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:43723 (local address 127.0.0.1:44982)
0807 03:14:24.163121 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:24.163142 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:24.163249 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:24.188157 (+ 24908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:24.188162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:24.188668 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:24.188877 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:24.188888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.193544 (+ 4656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.193551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:24.194984 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:24.194999 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.196143 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.196149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:24.196358 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.196977 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:24.196999 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:24.199703 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:24.206846 (+ 7143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:24.206852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:24.206857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:24.207394 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:24.207828 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:24.207833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:24.207837 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:24.208008 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:24.208709 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:24.208714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:24.208859 (+ 145us) client_negotiation.cc:769] Sending connection context
0807 03:14:24.209007 (+ 148us) client_negotiation.cc:240] Negotiation successful
0807 03:14:24.209299 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260807 03:14:24.210882 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.163466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.17.31.60:43723)
0807 03:14:24.187424 (+ 23958us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.187430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.187449 (+ 19us) server_negotiation.cc:407] Connection header received
0807 03:14:24.187497 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.187502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.187603 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.189033 (+ 1430us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.192266 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.193360 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.195174 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.195989 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.196122 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.199903 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:24.199909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:24.199913 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:24.199948 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:24.206682 (+ 6734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.207558 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.207563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.207566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.207702 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:24.208165 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:24.208172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:24.208176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:24.208555 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:24.210046 (+ 1491us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.210196 (+ 150us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.210466 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23838,"thread_start_us":28154,"threads_started":1}
I20260807 03:14:24.216457 8682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a: Bootstrap starting.
I20260807 03:14:24.220571 8682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:24.221989 8682 log.cc:824] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a: Log is configured to *not* fsync() on all Append() calls
I20260807 03:14:24.243031 8682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a: No bootstrap required, opened a new log
I20260807 03:14:24.244597 8765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 605ac0ec3e87491d93c2c7e3530ed2e1. Latest consensus state: current_term: 1 leader_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } } }
I20260807 03:14:24.244946 8765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:14:24.248538 8804 mvcc.cc:205] Tried to move back new op lower bound from 7315752813369511936 to 7315752813122310144. Current Snapshot: MvccSnapshot[applied={T|T < 7315752813369511936}]
I20260807 03:14:24.261862 8805 mvcc.cc:205] Tried to move back new op lower bound from 7315752813369511936 to 7315752813122310144. Current Snapshot: MvccSnapshot[applied={T|T < 7315752813369511936}]
I20260807 03:14:24.262563 8682 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } }
I20260807 03:14:24.263306 8682 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6fa76da8bcf4c30b1b55dcaf617911a, State: Initialized, Role: FOLLOWER
I20260807 03:14:24.264204 8682 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } }
I20260807 03:14:24.278936 8779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } } }
I20260807 03:14:24.279270 8779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:24.281883 8802 catalog_manager.cc:1347] Generated new cluster ID: 61ea97e000a14dc593e487878394a5f6
I20260807 03:14:24.281994 8802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:14:24.282646 8765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } } }
I20260807 03:14:24.282910 8765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:14:24.285231 8811 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } } }
I20260807 03:14:24.285753 8811 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:14:24.288051 8682 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:14:24.288467 8656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:24.290462 8657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a"
I20260807 03:14:24.293098 8656 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a [term 1 FOLLOWER]: Refusing update from remote peer 605ac0ec3e87491d93c2c7e3530ed2e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:14:24.296586 8657 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 605ac0ec3e87491d93c2c7e3530ed2e1 in term 1.
I20260807 03:14:24.298987 8794 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:14:24.324988 8802 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:14:24.326918 8802 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:14:24.329270 8811 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 605ac0ec3e87491d93c2c7e3530ed2e1. Latest consensus state: current_term: 1 leader_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } } }
I20260807 03:14:24.329593 8811 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:14:24.329705 8779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } } }
I20260807 03:14:24.330005 8779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1 [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:24.331085 8816 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "605ac0ec3e87491d93c2c7e3530ed2e1" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 46781 } } peers { permanent_uuid: "a6fa76da8bcf4c30b1b55dcaf617911a" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 36549 } } peers { permanent_uuid: "460527176d924a0faf42ee2f867546d9" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 43723 } } }
I20260807 03:14:24.332607 8816 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:14:24.350162 8802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 605ac0ec3e87491d93c2c7e3530ed2e1: Generated new TSK 0
I20260807 03:14:24.351418 8802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:14:24.361445 8828 catalog_manager.cc:1259] Loaded cluster ID: 61ea97e000a14dc593e487878394a5f6
I20260807 03:14:24.361616 8828 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a: loading cluster ID for follower catalog manager: success
I20260807 03:14:24.368582 8828 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a: acquiring CA information for follower catalog manager: success
I20260807 03:14:24.371063 8828 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a6fa76da8bcf4c30b1b55dcaf617911a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 03:14:24.488957 8686 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:24.476662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49590 (local address 127.17.31.61:36549)
0807 03:14:24.476909 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.476916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.486932 (+ 10016us) server_negotiation.cc:407] Connection header received
0807 03:14:24.487255 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.487261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.487347 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.487425 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.488708 (+ 1283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49590: BlockingRecv error: recv got EOF from 127.0.0.1:49590 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260807 03:14:24.491238 8762 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:24.478804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.17.31.60:43723)
0807 03:14:24.479021 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.479027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.489614 (+ 10587us) server_negotiation.cc:407] Connection header received
0807 03:14:24.489849 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.489857 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.489962 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.490034 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.490996 (+ 962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44984: BlockingRecv error: recv got EOF from 127.0.0.1:44984 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260807 03:14:24.493551 8610 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:24.487117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47398 (local address 127.17.31.62:46781)
0807 03:14:24.487324 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.487330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.491877 (+ 4547us) server_negotiation.cc:407] Connection header received
0807 03:14:24.492152 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.492158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.492244 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.492332 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.493302 (+ 970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47398: BlockingRecv error: recv got EOF from 127.0.0.1:47398 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 03:14:24.561455 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.538678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.17.31.61:36549)
0807 03:14:24.538914 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.538923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.553138 (+ 14215us) server_negotiation.cc:407] Connection header received
0807 03:14:24.553448 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.553454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.553690 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.553863 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:24.554571 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.555728 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.557707 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.558377 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.558488 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.559652 (+ 1164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:24.560346 (+ 694us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:24.560491 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.561020 (+ 529us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.561113 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260807 03:14:24.617007 8762 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:24.595188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44996 (local address 127.17.31.60:43723)
0807 03:14:24.595419 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.595426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.614940 (+ 19514us) server_negotiation.cc:407] Connection header received
0807 03:14:24.615214 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.615221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.615352 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.615463 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.616721 (+ 1258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44996: BlockingRecv error: recv got EOF from 127.0.0.1:44996 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 03:14:24.667122 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.652493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49606 (local address 127.17.31.61:36549)
0807 03:14:24.652702 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.652708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.659717 (+ 7009us) server_negotiation.cc:407] Connection header received
0807 03:14:24.660242 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.660249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.660394 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.660463 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:24.661182 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.662403 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.664575 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.665328 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.665449 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.666122 (+ 673us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:24.666329 (+ 207us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:24.666399 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.666671 (+ 272us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.666769 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260807 03:14:24.684437 8762 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:24.682138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45004 (local address 127.17.31.60:43723)
0807 03:14:24.682374 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.682380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.682879 (+ 499us) server_negotiation.cc:407] Connection header received
0807 03:14:24.683074 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.683080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.683176 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.683295 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.684197 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45004: BlockingRecv error: recv got EOF from 127.0.0.1:45004 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 03:14:24.701254 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.690364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.17.31.61:36549)
0807 03:14:24.690566 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.690572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.694874 (+ 4302us) server_negotiation.cc:407] Connection header received
0807 03:14:24.695126 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.695132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.695223 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.695325 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:24.695836 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.697071 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.699094 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.699813 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.699946 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.700421 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:24.700562 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:24.700653 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.700817 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.700907 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260807 03:14:24.720357 8762 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:24.715605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.17.31.60:43723)
0807 03:14:24.715803 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.715823 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.718859 (+ 3036us) server_negotiation.cc:407] Connection header received
0807 03:14:24.719021 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.719029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.719130 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.719251 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.720089 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45012: BlockingRecv error: recv got EOF from 127.0.0.1:45012 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 03:14:24.735035 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.728165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49622 (local address 127.17.31.61:36549)
0807 03:14:24.728356 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.728362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.728883 (+ 521us) server_negotiation.cc:407] Connection header received
0807 03:14:24.729116 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.729125 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.729249 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.729356 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:24.729831 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.731066 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.732951 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.733651 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.733797 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.734251 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:24.734413 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:24.734512 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.734645 (+ 133us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.734726 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260807 03:14:24.756436 8762 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:24.751839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45018 (local address 127.17.31.60:43723)
0807 03:14:24.752113 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.752119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.754919 (+ 2800us) server_negotiation.cc:407] Connection header received
0807 03:14:24.755120 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.755127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.755230 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.755344 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.756222 (+ 878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45018: BlockingRecv error: recv got EOF from 127.0.0.1:45018 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
I20260807 03:14:24.775233 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.768146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.17.31.61:36549)
0807 03:14:24.768336 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.768342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.768860 (+ 518us) server_negotiation.cc:407] Connection header received
0807 03:14:24.769076 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.769082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.769178 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.769288 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:24.769772 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.770975 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.772887 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.773623 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.773811 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.774282 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:24.774488 (+ 206us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:24.774578 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.774779 (+ 201us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.774867 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260807 03:14:24.787000 8762 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:24.784618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.17.31.60:43723)
0807 03:14:24.784825 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.784832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.785448 (+ 616us) server_negotiation.cc:407] Connection header received
0807 03:14:24.785640 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.785646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.785739 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.785851 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.786746 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45026: BlockingRecv error: recv got EOF from 127.0.0.1:45026 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 03:14:24.811547 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.806157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49642 (local address 127.17.31.61:36549)
0807 03:14:24.806363 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.806372 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.806759 (+ 387us) server_negotiation.cc:407] Connection header received
0807 03:14:24.806981 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.806986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.807058 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.807155 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:24.807582 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.808412 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.809724 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.810193 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.810343 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.810776 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:24.810912 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:24.811012 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.811191 (+ 179us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.811282 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260807 03:14:24.819324 8762 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:24.817442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45034 (local address 127.17.31.60:43723)
0807 03:14:24.817655 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.817661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.817979 (+ 318us) server_negotiation.cc:407] Connection header received
0807 03:14:24.818160 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.818165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.818277 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.818416 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.819091 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45034: BlockingRecv error: recv got EOF from 127.0.0.1:45034 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 03:14:24.861670 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.855601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49644 (local address 127.17.31.61:36549)
0807 03:14:24.855835 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.855841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.856178 (+ 337us) server_negotiation.cc:407] Connection header received
0807 03:14:24.856408 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.856413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.856478 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.856566 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:24.857085 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.857850 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.859886 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.860323 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.860409 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.860914 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:24.861011 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:24.861073 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.861240 (+ 167us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.861298 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260807 03:14:24.869396 8762 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:24.867541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.17.31.60:43723)
0807 03:14:24.867746 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.867752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.868050 (+ 298us) server_negotiation.cc:407] Connection header received
0807 03:14:24.868168 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.868173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.868294 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.868409 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.869152 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45042: BlockingRecv error: recv got EOF from 127.0.0.1:45042 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 03:14:24.942446 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:24.936490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49646 (local address 127.17.31.61:36549)
0807 03:14:24.936658 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.936663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.937114 (+ 451us) server_negotiation.cc:407] Connection header received
0807 03:14:24.937287 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.937293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.937376 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.937496 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:24.937854 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.939054 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.940580 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:24.941088 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:24.941223 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:24.941655 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:24.941784 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:24.941875 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:24.942054 (+ 179us) server_negotiation.cc:299] Negotiation successful
0807 03:14:24.942137 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260807 03:14:24.950700 8762 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:24.948815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.17.31.60:43723)
0807 03:14:24.948967 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:24.948971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:24.949276 (+ 305us) server_negotiation.cc:407] Connection header received
0807 03:14:24.949441 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:24.949445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:24.949512 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:24.949591 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:24.950487 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45052: BlockingRecv error: recv got EOF from 127.0.0.1:45052 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260807 03:14:25.050809 8776 catalog_manager.cc:1259] Loaded cluster ID: 61ea97e000a14dc593e487878394a5f6
I20260807 03:14:25.050941 8776 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9: loading cluster ID for follower catalog manager: success
I20260807 03:14:25.053683 8776 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9: acquiring CA information for follower catalog manager: success
I20260807 03:14:25.055492 8776 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 460527176d924a0faf42ee2f867546d9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 03:14:25.086843 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:25.082053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49662 (local address 127.17.31.61:36549)
0807 03:14:25.082340 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:25.082346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:25.082611 (+ 265us) server_negotiation.cc:407] Connection header received
0807 03:14:25.082788 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:25.082792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:25.082874 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:25.082948 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:25.083300 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:25.083973 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:25.085250 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:25.085672 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:25.085761 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:25.086115 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:25.086224 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:25.086287 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:25.086457 (+ 170us) server_negotiation.cc:299] Negotiation successful
0807 03:14:25.086536 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 03:14:25.097872 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:25.092633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.17.31.60:43723)
0807 03:14:25.092850 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:25.092855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:25.093084 (+ 229us) server_negotiation.cc:407] Connection header received
0807 03:14:25.093200 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:25.093204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:25.093355 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:25.093451 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:25.093784 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:25.094557 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:25.095784 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:25.096215 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:25.096317 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:25.096769 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:25.097324 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:25.097433 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:25.097510 (+ 77us) server_negotiation.cc:299] Negotiation successful
0807 03:14:25.097575 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 03:14:25.113215 8942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:25.103705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.17.31.62:46781)
0807 03:14:25.105831 (+ 2126us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:25.105839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:25.106293 (+ 454us) server_negotiation.cc:407] Connection header received
0807 03:14:25.106670 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:25.106676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:25.106915 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:25.107089 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:25.107883 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:25.108733 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:25.110758 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:25.111406 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:25.111530 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:25.112048 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:25.112523 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:25.112620 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:25.112797 (+ 177us) server_negotiation.cc:299] Negotiation successful
0807 03:14:25.112888 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1990,"thread_start_us":1888,"threads_started":1}
I20260807 03:14:25.113315 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:25.103685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45066 (local address 127.17.31.60:43723)
0807 03:14:25.103898 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:25.103905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:25.106940 (+ 3035us) server_negotiation.cc:407] Connection header received
0807 03:14:25.107097 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:25.107102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:25.107177 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:25.107264 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:25.108117 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:25.109191 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:25.111315 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:25.111841 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:25.112002 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:25.112564 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:25.112705 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:25.112787 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:25.112988 (+ 201us) server_negotiation.cc:299] Negotiation successful
0807 03:14:25.113072 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 03:14:25.113945 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:25.104036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.17.31.61:36549)
0807 03:14:25.104214 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:25.104221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:25.106561 (+ 2340us) server_negotiation.cc:407] Connection header received
0807 03:14:25.106875 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:25.106884 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:25.106974 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:25.107066 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:25.108235 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:25.109317 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:25.111188 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:25.111803 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:25.111943 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:25.113185 (+ 1242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:25.113314 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:25.113382 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:25.113606 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 03:14:25.113682 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 03:14:25.118672 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 8536
I20260807 03:14:25.119850 8600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:14:25.232398 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 8536
I20260807 03:14:25.255036 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 8613
I20260807 03:14:25.256115 8677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:14:25.363478 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 8613
I20260807 03:14:25.385865 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 8689
I20260807 03:14:25.387091 8754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:14:25.494647 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 8689
2026-08-07T03:14:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4406 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 03:14:25 dist-test-slave-tm1g krb5kdc[8949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 03:14:25 dist-test-slave-tm1g krb5kdc[8949](info): set up 2 sockets
Aug 07 03:14:25 dist-test-slave-tm1g krb5kdc[8949](info): commencing operation
krb5kdc: starting...
W20260807 03:14:26.628604 17532 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.001s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/test-user.keytab.
Aug 07 03:14:26 dist-test-slave-tm1g krb5kdc[8949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T03:14:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T03:14:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.31.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
Entry for principal kudu/127.17.31.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab.
I20260807 03:14:26.916852 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:36269
--webserver_interface=127.17.31.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:38309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:36269,127.17.31.61:42649,127.17.31.60:44965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 03:14:27.243665 8963 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 03:14:27.243993 8963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:27.244118 8963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:27.244237 8963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:14:27.251849 8963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:27.251927 8963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:14:27.252039 8963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:27.252125 8963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:27.252230 8963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:27.261560 8963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:38309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:36269,127.17.31.61:42649,127.17.31.60:44965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.62.keytab
--principal=kudu/127.17.31.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.62:36269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.31.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:27.262825 8963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:27.264092 8963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:14:27.275656 8968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:27.275591 8969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:27.275528 8971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:27.276741 8963 server_base.cc:1034] running on GCE node
I20260807 03:14:27.277504 8963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:27.278903 8963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:27.280124 8963 hybrid_clock.cc:648] HybridClock initialized: now 1786072467280072 us; error 35 us; skew 500 ppm
I20260807 03:14:27.280440 8963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:27 dist-test-slave-tm1g krb5kdc[8949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072467, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:27.286952 8963 init.cc:377] Logged in from keytab as kudu/127.17.31.62@KRBTEST.COM (short username kudu)
I20260807 03:14:27.288311 8963 webserver.cc:460] Webserver started at http://127.17.31.62:39633/ using document root <none> and password file <none>
I20260807 03:14:27.288959 8963 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:27.289106 8963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:27.289431 8963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:27.292320 8963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/data/instance:
uuid: "d06d4f1044b3403fa78eb991913db1fc"
format_stamp: "Formatted at 2026-08-07 03:14:27 on dist-test-slave-tm1g"
server_key: "a439c5489d6672e762cd99145ca61ce3"
server_key_iv: "7965cf55d7b9266de626c7e9c292fa0b"
server_key_version: "encryptionkey@0"
I20260807 03:14:27.293109 8963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/wal/instance:
uuid: "d06d4f1044b3403fa78eb991913db1fc"
format_stamp: "Formatted at 2026-08-07 03:14:27 on dist-test-slave-tm1g"
server_key: "a439c5489d6672e762cd99145ca61ce3"
server_key_iv: "7965cf55d7b9266de626c7e9c292fa0b"
server_key_version: "encryptionkey@0"
I20260807 03:14:27.298082 8963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:14:27.301419 8978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:27.302742 8963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:14:27.302925 8963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/wal
uuid: "d06d4f1044b3403fa78eb991913db1fc"
format_stamp: "Formatted at 2026-08-07 03:14:27 on dist-test-slave-tm1g"
server_key: "a439c5489d6672e762cd99145ca61ce3"
server_key_iv: "7965cf55d7b9266de626c7e9c292fa0b"
server_key_version: "encryptionkey@0"
I20260807 03:14:27.303139 8963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:27.332729 8963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:27.336318 8963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:27.336591 8963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:27.354779 8963 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.62:36269
I20260807 03:14:27.354813 9032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.62:36269 every 8 connection(s)
I20260807 03:14:27.356051 8963 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/data/info.pb
I20260807 03:14:27.359486 9033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:27.362372 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 8963
I20260807 03:14:27.362576 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-0/wal/instance
I20260807 03:14:27.363154 17532 external_mini_cluster.cc:1428] Setting key 8e13ef62b74c58cd48e7b33e768c36c9
I20260807 03:14:27.367951 9033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:14:27 dist-test-slave-tm1g krb5kdc[8949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072467, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:14:27.393369 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.371585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:36269 (local address 127.0.0.1:48616)
0807 03:14:27.372774 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.372821 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:27.375487 (+ 2666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:27.376301 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:27.376314 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:27.377099 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:27.377616 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:27.377642 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:27.380021 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:27.380025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:27.381077 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:27.381088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:27.382448 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:27.382454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:27.382571 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:27.383217 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:27.383237 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:27.385246 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:27.389655 (+ 4409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:27.389665 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:27.389673 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:27.390026 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:27.390482 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:27.390485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:27.390488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:27.390632 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:27.391565 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:27.391574 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:27.391752 (+ 178us) client_negotiation.cc:769] Sending connection context
0807 03:14:27.391956 (+ 204us) client_negotiation.cc:240] Negotiation successful
0807 03:14:27.392384 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"spinlock_wait_cycles":27520,"thread_start_us":721,"threads_started":1}
I20260807 03:14:27.393411 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.371586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.17.31.62:36269)
0807 03:14:27.372488 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:27.372511 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:27.372954 (+ 443us) server_negotiation.cc:407] Connection header received
0807 03:14:27.375832 (+ 2878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:27.375837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:27.376063 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:27.376212 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:27.377845 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:27.379876 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:27.381217 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:27.382184 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:27.382411 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:27.385496 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:27.385508 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:27.385525 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:27.385567 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:27.389520 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:27.390148 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:27.390158 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:27.390164 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:27.390366 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:27.390744 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:27.390752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:27.390757 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:27.391431 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:27.391631 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:27.392147 (+ 516us) server_negotiation.cc:299] Negotiation successful
0807 03:14:27.392583 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":635,"thread_start_us":409,"threads_started":1}
I20260807 03:14:27.396356 9033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.17.31.61@KRBTEST.COM; defaulting to no policy
I20260807 03:14:27.397399 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.397074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:42649 (local address 127.0.0.1:36460)
0807 03:14:27.397208 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.397298 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:42649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 03:14:27.397732 8980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.31.61:42649: connect: Connection refused (error 111)
I20260807 03:14:27.399133 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.398775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:42649 (local address 127.0.0.1:36472)
0807 03:14:27.398898 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.398991 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:42649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 03:14:27.399379 9033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.61:42649: Network error: Client connection negotiation failed: client connection to 127.17.31.61:42649: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.61.keytab.
Entry for principal kudu/127.17.31.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.61.keytab.
I20260807 03:14:27.429319 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.61:42649
--webserver_interface=127.17.31.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.61.keytab
--principal=kudu/127.17.31.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:38309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:36269,127.17.31.61:42649,127.17.31.60:44965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:14:27.448693 9033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } attempt: 1
I20260807 03:14:27.449911 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.449423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:42649 (local address 127.0.0.1:36480)
0807 03:14:27.449660 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.449770 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:42649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 03:14:27.450914 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.450568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:42649 (local address 127.0.0.1:36484)
0807 03:14:27.450717 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.450790 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:42649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 03:14:27.451092 9033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.61:42649: Network error: Client connection negotiation failed: client connection to 127.17.31.61:42649: connect: Connection refused (error 111)
I20260807 03:14:27.519330 9033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } attempt: 2
I20260807 03:14:27.520545 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.520031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:42649 (local address 127.0.0.1:36490)
0807 03:14:27.520303 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.520397 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:42649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 03:14:27.521670 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.521296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:42649 (local address 127.0.0.1:36492)
0807 03:14:27.521468 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.521548 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:42649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260807 03:14:27.521888 9033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.61:42649: Network error: Client connection negotiation failed: client connection to 127.17.31.61:42649: connect: Connection refused (error 111)
I20260807 03:14:27.613137 9033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } attempt: 3
I20260807 03:14:27.614244 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.613805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:42649 (local address 127.0.0.1:36504)
0807 03:14:27.614007 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.614102 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:42649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260807 03:14:27.615331 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.615000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:42649 (local address 127.0.0.1:36514)
0807 03:14:27.615144 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.615205 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:42649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260807 03:14:27.615504 9033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.61:42649: Network error: Client connection negotiation failed: client connection to 127.17.31.61:42649: connect: Connection refused (error 111)
I20260807 03:14:27.758734 9033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } attempt: 4
I20260807 03:14:27.759897 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.759380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:42649 (local address 127.0.0.1:36522)
0807 03:14:27.759684 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.759765 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:42649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260807 03:14:27.760890 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.760549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:42649 (local address 127.0.0.1:36528)
0807 03:14:27.760699 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.760771 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.61:42649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 03:14:27.761057 9033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.61:42649: Network error: Client connection negotiation failed: client connection to 127.17.31.61:42649: connect: Connection refused (error 111)
W20260807 03:14:27.770615 9040 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 03:14:27.770962 9040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:27.771091 9040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:27.771220 9040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 03:14:27.778751 9040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:27.778891 9040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 03:14:27.779009 9040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:27.779096 9040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:27.779213 9040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:27.789053 9040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:38309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:36269,127.17.31.61:42649,127.17.31.60:44965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.61.keytab
--principal=kudu/127.17.31.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.61:42649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.31.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:27.790280 9040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:27.791687 9040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:14:27.803153 9046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:27.803130 9047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:27.803192 9049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:27.804095 9040 server_base.cc:1034] running on GCE node
I20260807 03:14:27.804782 9040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:27.806110 9040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:27.807363 9040 hybrid_clock.cc:648] HybridClock initialized: now 1786072467807288 us; error 55 us; skew 500 ppm
I20260807 03:14:27.807690 9040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:27 dist-test-slave-tm1g krb5kdc[8949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072467, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:27.810704 9040 init.cc:377] Logged in from keytab as kudu/127.17.31.61@KRBTEST.COM (short username kudu)
I20260807 03:14:27.812047 9040 webserver.cc:460] Webserver started at http://127.17.31.61:40285/ using document root <none> and password file <none>
I20260807 03:14:27.812724 9040 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:27.812857 9040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:27.813166 9040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:27.815996 9040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/data/instance:
uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7"
format_stamp: "Formatted at 2026-08-07 03:14:27 on dist-test-slave-tm1g"
server_key: "a09c2e5a926a8e3ae005c96910a84f39"
server_key_iv: "be5640574189857d23807a74b8404995"
server_key_version: "encryptionkey@0"
I20260807 03:14:27.816771 9040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/wal/instance:
uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7"
format_stamp: "Formatted at 2026-08-07 03:14:27 on dist-test-slave-tm1g"
server_key: "a09c2e5a926a8e3ae005c96910a84f39"
server_key_iv: "be5640574189857d23807a74b8404995"
server_key_version: "encryptionkey@0"
I20260807 03:14:27.821662 9040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 03:14:27.824913 9056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:27.826092 9040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260807 03:14:27.826272 9040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/wal
uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7"
format_stamp: "Formatted at 2026-08-07 03:14:27 on dist-test-slave-tm1g"
server_key: "a09c2e5a926a8e3ae005c96910a84f39"
server_key_iv: "be5640574189857d23807a74b8404995"
server_key_version: "encryptionkey@0"
I20260807 03:14:27.826537 9040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:27.864655 9040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:27.868448 9040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:27.868726 9040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:27.886556 9040 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.61:42649
I20260807 03:14:27.886591 9111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.61:42649 every 8 connection(s)
I20260807 03:14:27.887810 9040 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/data/info.pb
I20260807 03:14:27.891331 9112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:27.894249 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 9040
I20260807 03:14:27.894490 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-1/wal/instance
I20260807 03:14:27.895030 17532 external_mini_cluster.cc:1428] Setting key 8ab60470b840a410ca2fe3433a826513
I20260807 03:14:27.899727 9112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:14:27 dist-test-slave-tm1g krb5kdc[8949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072467, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.61@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:14:27.922971 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.902988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48630 (local address 127.17.31.62:36269)
0807 03:14:27.903474 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:27.903481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:27.904474 (+ 993us) server_negotiation.cc:407] Connection header received
0807 03:14:27.907518 (+ 3044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:27.907522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:27.907649 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:27.907840 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:27.909555 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:27.910787 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:27.912249 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:27.912981 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:27.913119 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:27.916978 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:27.916983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:27.916986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:27.917029 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:27.919729 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:27.920697 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:27.920700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:27.920702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:27.920815 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:27.921258 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:27.921261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:27.921263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:27.921524 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:27.921638 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:27.922535 (+ 897us) server_negotiation.cc:299] Negotiation successful
0807 03:14:27.922709 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":239,"threads_started":1}
I20260807 03:14:27.924544 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.903488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:36269 (local address 127.0.0.1:48630)
0807 03:14:27.904289 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.904349 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:27.907232 (+ 2883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:27.908051 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:27.908064 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:27.908828 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:27.909343 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:27.909360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:27.910947 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:27.910951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:27.912106 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:27.912118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:27.913171 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:27.913176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:27.913365 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:27.914017 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:27.914039 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:27.916738 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:27.919897 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:27.919908 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:27.919924 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:27.920582 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:27.920917 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:27.920924 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:27.920930 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:27.921154 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:27.921651 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:27.921662 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:27.922118 (+ 456us) client_negotiation.cc:769] Sending connection context
0807 03:14:27.922451 (+ 333us) client_negotiation.cc:240] Negotiation successful
0807 03:14:27.922994 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":311,"threads_started":1}
I20260807 03:14:27.926442 9112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.17.31.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.31.60@KRBTEST.COM" created.
Aug 07 03:14:27 dist-test-slave-tm1g krb5kdc[8949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072467, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.61@KRBTEST.COM for kudu/127.17.31.61@KRBTEST.COM
I20260807 03:14:27.942492 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.927255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:42649 (local address 127.0.0.1:36542)
0807 03:14:27.927492 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.927515 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:27.927621 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:27.928534 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:27.928537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:27.928914 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:27.929048 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:27.929056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:27.930742 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:27.930747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:27.931570 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:27.931581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:27.932821 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:27.932826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:27.932941 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:27.933503 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:27.933519 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:27.935418 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:27.939625 (+ 4207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:27.939632 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:27.939638 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:27.940114 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:27.940538 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:27.940545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:27.940550 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:27.940703 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:27.941487 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:27.941491 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:27.941620 (+ 129us) client_negotiation.cc:769] Sending connection context
0807 03:14:27.941759 (+ 139us) client_negotiation.cc:240] Negotiation successful
0807 03:14:27.942027 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260807 03:14:27.942597 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.927484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.17.31.61:42649)
0807 03:14:27.928148 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:27.928153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:27.928179 (+ 26us) server_negotiation.cc:407] Connection header received
0807 03:14:27.928219 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:27.928225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:27.928388 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:27.928501 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:27.929243 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:27.930589 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:27.931715 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:27.932664 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:27.932826 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:27.935629 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:27.935644 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:27.935661 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:27.935708 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:27.939491 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:27.940258 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:27.940269 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:27.940275 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:27.940425 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:27.940826 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:27.940834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:27.940839 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:27.941307 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:27.941507 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:27.941903 (+ 396us) server_negotiation.cc:299] Negotiation successful
0807 03:14:27.942146 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":520,"thread_start_us":389,"threads_started":1}
I20260807 03:14:27.944929 9112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:14:27.945956 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.945536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45002)
0807 03:14:27.945695 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.945831 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 03:14:27.946179 9062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
I20260807 03:14:27.947932 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.947579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45004)
0807 03:14:27.947738 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.947825 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260807 03:14:27.948148 9112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:44965: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.31.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.60.keytab.
Entry for principal kudu/127.17.31.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.60.keytab.
I20260807 03:14:27.963441 17532 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
/tmp/dist-test-tasko9xIn4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.60:44965
--webserver_interface=127.17.31.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.60.keytab
--principal=kudu/127.17.31.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.31.20:38309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.31.62:36269,127.17.31.61:42649,127.17.31.60:44965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 03:14:27.997548 9112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } attempt: 1
I20260807 03:14:27.998759 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.998286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45008)
0807 03:14:27.998519 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.998619 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260807 03:14:27.999833 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:27.999490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45016)
0807 03:14:27.999656 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:27.999728 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 03:14:28.000037 9112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:44965: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
I20260807 03:14:28.060310 9033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } attempt: 5
I20260807 03:14:28.068230 9112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } attempt: 2
Aug 07 03:14:28 dist-test-slave-tm1g krb5kdc[8949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072467, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for kudu/127.17.31.61@KRBTEST.COM
I20260807 03:14:28.069126 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.068738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45030)
0807 03:14:28.068900 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.068984 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 03:14:28.070459 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.069928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45040)
0807 03:14:28.070177 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.070267 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260807 03:14:28.070670 9112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:44965: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
I20260807 03:14:28.075594 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.061123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:42649 (local address 127.0.0.1:36544)
0807 03:14:28.061353 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.061377 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:28.061499 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:28.061944 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:28.061947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:28.062367 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:28.062553 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:28.062562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.064059 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.064062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:28.064938 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:28.064949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.065934 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.065938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:28.066108 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.066747 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:28.066765 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:28.069086 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:28.073156 (+ 4070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:28.073159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:28.073162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:28.073728 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:28.074131 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:28.074134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:28.074136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:28.074262 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:28.074900 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:28.074903 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:28.075039 (+ 136us) client_negotiation.cc:769] Sending connection context
0807 03:14:28.075176 (+ 137us) client_negotiation.cc:240] Negotiation successful
0807 03:14:28.075368 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260807 03:14:28.076020 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.061297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36544 (local address 127.17.31.61:42649)
0807 03:14:28.061512 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.061520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.061580 (+ 60us) server_negotiation.cc:407] Connection header received
0807 03:14:28.061674 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.061682 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.061780 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.061900 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.062717 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.063914 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.065095 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.065796 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.065932 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.069285 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:28.069291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:28.069296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:28.069335 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:28.073026 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.073854 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.073859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.073863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.074026 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.074411 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.074416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.074419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.074780 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:28.074966 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:28.075368 (+ 402us) server_negotiation.cc:299] Negotiation successful
0807 03:14:28.075627 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 03:14:28.077065 9033 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } has no permanent_uuid. Determining permanent_uuid...
I20260807 03:14:28.078025 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.077666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45044)
0807 03:14:28.077813 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.077881 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 03:14:28.078929 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.078616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45058)
0807 03:14:28.078756 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.078826 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 03:14:28.079125 9033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:44965: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
I20260807 03:14:28.130393 9033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } attempt: 1
I20260807 03:14:28.131639 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.131180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45074)
0807 03:14:28.131375 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.131464 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 03:14:28.132616 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.132305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45088)
0807 03:14:28.132455 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.132520 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 03:14:28.132822 9033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:44965: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
I20260807 03:14:28.161906 9112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } attempt: 3
I20260807 03:14:28.163488 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.162649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45096)
0807 03:14:28.163235 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.163321 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355}
I20260807 03:14:28.164578 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.164140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45106)
0807 03:14:28.164372 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.164466 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260807 03:14:28.164826 9112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:44965: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
I20260807 03:14:28.201023 9033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } attempt: 2
I20260807 03:14:28.202168 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.201695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45112)
0807 03:14:28.201912 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.202007 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 03:14:28.203187 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.202867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45128)
0807 03:14:28.203016 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.203077 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 03:14:28.203397 9033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:44965: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
W20260807 03:14:28.301367 9120 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 03:14:28.301735 9120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 03:14:28.301867 9120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 03:14:28.302008 9120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260807 03:14:28.308032 9112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } attempt: 4
I20260807 03:14:28.309242 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.308656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45134)
0807 03:14:28.308912 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.309026 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260807 03:14:28.309589 9120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 03:14:28.309682 9120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260807 03:14:28.309607 9033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } attempt: 3
W20260807 03:14:28.309823 9120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 03:14:28.309875 9120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 03:14:28.309952 9120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 03:14:28.310369 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.309944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45140)
0807 03:14:28.310102 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.310202 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260807 03:14:28.310601 9112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:44965: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
I20260807 03:14:28.310760 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.310212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45146)
0807 03:14:28.310454 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.310554 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 03:14:28.311838 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.311431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:45160)
0807 03:14:28.311661 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.311742 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260807 03:14:28.312114 9033 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.31.60:44965: Network error: Client connection negotiation failed: client connection to 127.17.31.60:44965: connect: Connection refused (error 111)
I20260807 03:14:28.320056 9120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.31.20:38309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.31.62:36269,127.17.31.61:42649,127.17.31.60:44965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/krb5kdc/kudu_127.17.31.60.keytab
--principal=kudu/127.17.31.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.31.60:44965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.31.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 02:01:27 UTC on ccycloud.quasar-hkknko.root.comops.site
build id 4571739
ASAN enabled
I20260807 03:14:28.321277 9120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 03:14:28.322643 9120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 03:14:28.334424 9125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:28.334554 9128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 03:14:28.334455 9126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 03:14:28.335649 9120 server_base.cc:1034] running on GCE node
I20260807 03:14:28.336359 9120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 03:14:28.337730 9120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 03:14:28.339013 9120 hybrid_clock.cc:648] HybridClock initialized: now 1786072468338934 us; error 44 us; skew 500 ppm
I20260807 03:14:28.339339 9120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 07 03:14:28 dist-test-slave-tm1g krb5kdc[8949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072468, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 03:14:28.342665 9120 init.cc:377] Logged in from keytab as kudu/127.17.31.60@KRBTEST.COM (short username kudu)
I20260807 03:14:28.344046 9120 webserver.cc:460] Webserver started at http://127.17.31.60:41205/ using document root <none> and password file <none>
I20260807 03:14:28.344729 9120 fs_manager.cc:359] Metadata directory not provided
I20260807 03:14:28.344872 9120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 03:14:28.345199 9120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 03:14:28.348253 9120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/data/instance:
uuid: "6403fe944af544faa18018989027b054"
format_stamp: "Formatted at 2026-08-07 03:14:28 on dist-test-slave-tm1g"
server_key: "c98650ed564ea057afb4b6d6ad7434bf"
server_key_iv: "6fcbb4c95af13f7e3255c6025e3d4109"
server_key_version: "encryptionkey@0"
I20260807 03:14:28.349059 9120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/wal/instance:
uuid: "6403fe944af544faa18018989027b054"
format_stamp: "Formatted at 2026-08-07 03:14:28 on dist-test-slave-tm1g"
server_key: "c98650ed564ea057afb4b6d6ad7434bf"
server_key_iv: "6fcbb4c95af13f7e3255c6025e3d4109"
server_key_version: "encryptionkey@0"
I20260807 03:14:28.354195 9120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 03:14:28.357640 9135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 03:14:28.359032 9120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 03:14:28.359221 9120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/data,/tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/wal
uuid: "6403fe944af544faa18018989027b054"
format_stamp: "Formatted at 2026-08-07 03:14:28 on dist-test-slave-tm1g"
server_key: "c98650ed564ea057afb4b6d6ad7434bf"
server_key_iv: "6fcbb4c95af13f7e3255c6025e3d4109"
server_key_version: "encryptionkey@0"
I20260807 03:14:28.359438 9120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 03:14:28.392566 9120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 03:14:28.396953 9120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 03:14:28.397249 9120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 03:14:28.415567 9120 rpc_server.cc:307] RPC server started. Bound to: 127.17.31.60:44965
I20260807 03:14:28.415586 9191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.31.60:44965 every 8 connection(s)
I20260807 03:14:28.416915 9120 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/data/info.pb
I20260807 03:14:28.418639 17532 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu as pid 9120
I20260807 03:14:28.418840 17532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko9xIn4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786072106928531-17532-0/minicluster-data/master-2/wal/instance
I20260807 03:14:28.419297 17532 external_mini_cluster.cc:1428] Setting key e3ac7ac77c648a7d859e9cfc875e1e95
I20260807 03:14:28.421060 9192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 03:14:28.429603 9192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:14:28 dist-test-slave-tm1g krb5kdc[8949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.60@KRBTEST.COM
Aug 07 03:14:28 dist-test-slave-tm1g krb5kdc[8949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072468, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.60@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
I20260807 03:14:28.442247 9196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.420988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.17.31.60:44965)
0807 03:14:28.421781 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.421792 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.421832 (+ 40us) server_negotiation.cc:407] Connection header received
0807 03:14:28.423444 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.423463 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.423830 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.424147 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.425953 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.427324 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.429073 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.429707 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.429912 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.434519 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:28.434529 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:28.434541 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:28.434585 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:28.438134 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.438909 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.438915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.438921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.439103 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.439468 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.439477 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.439479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.440057 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:28.440371 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:28.440657 (+ 286us) server_negotiation.cc:299] Negotiation successful
0807 03:14:28.441112 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":390,"threads_started":1}
I20260807 03:14:28.447191 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.432320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.17.31.62:36269)
0807 03:14:28.432881 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.432890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.433433 (+ 543us) server_negotiation.cc:407] Connection header received
0807 03:14:28.433534 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.433541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.433626 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.433733 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.434956 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.435657 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.437709 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.438112 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.438203 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.441586 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:28.441590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:28.441594 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:28.441625 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:28.444311 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.445010 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.445017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.445023 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.445192 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.445645 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.445649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.445652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.446041 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:28.446226 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:28.446754 (+ 528us) server_negotiation.cc:299] Negotiation successful
0807 03:14:28.446922 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":253,"threads_started":1}
I20260807 03:14:28.447407 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.432313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.62:36269 (local address 127.0.0.1:44632)
0807 03:14:28.433233 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.433273 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:28.433425 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:28.433779 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:28.433784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:28.434430 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:28.434813 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:28.434825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.435815 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.435821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:28.437560 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:28.437573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.438329 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.438337 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:28.438477 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.439045 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:28.439063 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:28.441399 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:28.444449 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:28.444460 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:28.444466 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:28.444871 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:28.445358 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:28.445366 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:28.445371 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:28.445539 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:28.446205 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:28.446212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:28.446439 (+ 227us) client_negotiation.cc:769] Sending connection context
0807 03:14:28.446673 (+ 234us) client_negotiation.cc:240] Negotiation successful
0807 03:14:28.446999 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":538,"threads_started":1}
I20260807 03:14:28.448750 9192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:14:28 dist-test-slave-tm1g krb5kdc[8949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.62@KRBTEST.COM
Aug 07 03:14:28 dist-test-slave-tm1g krb5kdc[8949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072468, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.60@KRBTEST.COM for kudu/127.17.31.61@KRBTEST.COM
I20260807 03:14:28.460695 9199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.445047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.17.31.62:36269)
0807 03:14:28.445714 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.445721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.445745 (+ 24us) server_negotiation.cc:407] Connection header received
0807 03:14:28.445789 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.445793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.445855 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.445959 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.447601 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.448667 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.450257 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.450742 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.450874 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.454904 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:28.454911 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:28.454916 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:28.454953 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:28.457904 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.458704 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.458707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.458709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.458821 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.459222 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.459225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.459227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.459486 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:28.459756 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:28.460013 (+ 257us) server_negotiation.cc:299] Negotiation successful
0807 03:14:28.460243 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":320,"threads_started":1}
I20260807 03:14:28.462461 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.449511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.61:42649 (local address 127.0.0.1:54848)
0807 03:14:28.449681 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.449700 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:28.449784 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:28.450090 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:28.450095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:28.450534 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:28.450661 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:28.450668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.451974 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.451980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:28.452819 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:28.452829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.453541 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.453547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:28.453720 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.454226 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:28.454241 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:28.456331 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:28.459639 (+ 3308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:28.459645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:28.459648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:28.460141 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:28.460602 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:28.460606 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:28.460610 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:28.460801 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:28.461487 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:28.461490 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:28.461662 (+ 172us) client_negotiation.cc:769] Sending connection context
0807 03:14:28.461835 (+ 173us) client_negotiation.cc:240] Negotiation successful
0807 03:14:28.462110 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 03:14:28.463189 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.449543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54848 (local address 127.17.31.61:42649)
0807 03:14:28.449734 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.449738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.449754 (+ 16us) server_negotiation.cc:407] Connection header received
0807 03:14:28.449897 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.449901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.449961 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.450036 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.450803 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.451826 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.452953 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.453376 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.453529 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.456514 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:28.456518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:28.456521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:28.456552 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:28.459495 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.460284 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.460289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.460292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.460452 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.460985 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.460988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.460991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.461359 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:28.461527 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:28.461933 (+ 406us) server_negotiation.cc:299] Negotiation successful
0807 03:14:28.462134 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 03:14:28.464113 9192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 03:14:28 dist-test-slave-tm1g krb5kdc[8949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072468, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.60@KRBTEST.COM for kudu/127.17.31.60@KRBTEST.COM
Aug 07 03:14:28 dist-test-slave-tm1g krb5kdc[8949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.31.61@KRBTEST.COM
I20260807 03:14:28.477690 9196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.464899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.17.31.60:44965)
0807 03:14:28.465066 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.465071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.465086 (+ 15us) server_negotiation.cc:407] Connection header received
0807 03:14:28.465131 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.465134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.465180 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.465246 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.466071 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.467065 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.468460 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.469127 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.469244 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.472231 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:28.472238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:28.472243 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:28.472282 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:28.475010 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.475732 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.475735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.475737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.475853 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.476253 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.476256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.476258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.476618 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:28.476765 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:28.477109 (+ 344us) server_negotiation.cc:299] Negotiation successful
0807 03:14:28.477321 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 03:14:28.477917 9198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.464711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:48100)
0807 03:14:28.464889 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.464907 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:28.465000 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:28.465313 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:28.465318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:28.465753 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:28.465943 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:28.465955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.467185 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.467189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:28.468282 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:28.468298 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.469293 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.469298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:28.469428 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.469953 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:28.469968 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:28.472069 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:28.475140 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:28.475145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:28.475149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:28.475629 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:28.475957 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:28.475964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:28.475969 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:28.476149 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:28.476734 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:28.476737 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:28.476854 (+ 117us) client_negotiation.cc:769] Sending connection context
0807 03:14:28.477016 (+ 162us) client_negotiation.cc:240] Negotiation successful
0807 03:14:28.477472 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":165}
I20260807 03:14:28.479142 9200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.462386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54850 (local address 127.17.31.61:42649)
0807 03:14:28.463218 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.463221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.463264 (+ 43us) server_negotiation.cc:407] Connection header received
0807 03:14:28.463318 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.463325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.463395 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.463535 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.465060 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.466223 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.467718 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.468357 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.468480 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.473076 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:28.473085 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:28.473092 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:28.473131 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:28.476256 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.476997 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.477004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.477007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.477158 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.477591 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.477596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.477600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.477924 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:28.478204 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:28.478656 (+ 452us) server_negotiation.cc:299] Negotiation successful
0807 03:14:28.478812 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":545,"threads_started":1}
I20260807 03:14:28.483238 9192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054: Bootstrap starting.
I20260807 03:14:28.483846 17532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 03:14:28.486559 9192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:28.487850 9192 log.cc:824] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054: Log is configured to *not* fsync() on all Append() calls
I20260807 03:14:28.489324 9033 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } attempt: 4
I20260807 03:14:28.486146 9007 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d06d4f1044b3403fa78eb991913db1fc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44646
I20260807 03:14:28.491091 9192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054: No bootstrap required, opened a new log
Aug 07 03:14:28 dist-test-slave-tm1g krb5kdc[8949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072467, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.62@KRBTEST.COM for kudu/127.17.31.60@KRBTEST.COM
I20260807 03:14:28.502424 9192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } }
I20260807 03:14:28.502851 9192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6403fe944af544faa18018989027b054, State: Initialized, Role: FOLLOWER
I20260807 03:14:28.503021 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.489881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:48102)
0807 03:14:28.490047 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.490067 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:28.490172 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:28.490662 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:28.490669 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:28.491030 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:28.491243 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:28.491253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.492308 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.492314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:28.493282 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:28.493293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.494112 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.494118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:28.494405 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.494997 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:28.495015 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:28.497223 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:28.500490 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:28.500495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:28.500499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:28.500870 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:28.501252 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:28.501257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:28.501261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:28.501406 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:28.501957 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:28.501962 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:28.502137 (+ 175us) client_negotiation.cc:769] Sending connection context
0807 03:14:28.502278 (+ 141us) client_negotiation.cc:240] Negotiation successful
0807 03:14:28.502603 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260807 03:14:28.503021 9196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.490180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.17.31.60:44965)
0807 03:14:28.490419 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.490423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.490437 (+ 14us) server_negotiation.cc:407] Connection header received
0807 03:14:28.490475 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.490478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.490528 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.490627 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.491421 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.492176 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.493448 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.493925 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.494090 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.497436 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:28.497440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:28.497443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:28.497471 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:28.500334 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.500985 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.500992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.500996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.501132 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.501515 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.501520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.501524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.501837 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:28.501996 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:28.502424 (+ 428us) server_negotiation.cc:299] Negotiation successful
0807 03:14:28.502624 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 03:14:28.503425 9192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } }
I20260807 03:14:28.504748 9202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } } }
I20260807 03:14:28.505612 9202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:14:28.505915 9192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:14:28.507586 9033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc: Bootstrap starting.
I20260807 03:14:28.511085 9033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:28.512295 9033 log.cc:824] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc: Log is configured to *not* fsync() on all Append() calls
I20260807 03:14:28.516696 9033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc: No bootstrap required, opened a new log
W20260807 03:14:28.517643 9213 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 03:14:28.517755 9213 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 03:14:28.524181 9033 raft_consensus.cc:348] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } }
I20260807 03:14:28.524557 9033 raft_consensus.cc:729] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06d4f1044b3403fa78eb991913db1fc, State: Initialized, Role: FOLLOWER
I20260807 03:14:28.525045 9033 consensus_queue.cc:260] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } }
I20260807 03:14:28.515105 9120 master_runner.cc:186] Error getting master registration for 127.17.31.62:36269: OK, instance_id { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" instance_seqno: 1786072467349777 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 03:14:28.526589 9120 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 03:14:28.526252 9216 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } } }
I20260807 03:14:28.527246 9216 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:14:28.527529 9033 sys_catalog.cc:564] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:14:28.527619 9007 raft_consensus.cc:482] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 03:14:28.527832 9007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:28.531509 9007 raft_consensus.cc:504] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } }
I20260807 03:14:28.533372 9007 leader_election.cc:288] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [CANDIDATE]: Term 1 election: Requested vote from peers cf7cde467b534aaf9e4c5e4e9f0a86b7 (127.17.31.61:42649), 6403fe944af544faa18018989027b054 (127.17.31.60:44965)
I20260807 03:14:28.535138 9164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d06d4f1044b3403fa78eb991913db1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6403fe944af544faa18018989027b054"
I20260807 03:14:28.535725 9164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:28.540395 9164 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d06d4f1044b3403fa78eb991913db1fc in term 1.
I20260807 03:14:28.541242 8982 leader_election.cc:302] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6403fe944af544faa18018989027b054, d06d4f1044b3403fa78eb991913db1fc; no voters:
I20260807 03:14:28.541796 9216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 03:14:28.553887 9216 raft_consensus.cc:686] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [term 1 LEADER]: Becoming Leader. State: Replica: d06d4f1044b3403fa78eb991913db1fc, State: Running, Role: LEADER
I20260807 03:14:28.555375 9216 consensus_queue.cc:237] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } }
I20260807 03:14:28.564756 9227 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d06d4f1044b3403fa78eb991913db1fc. Latest consensus state: current_term: 1 leader_uuid: "d06d4f1044b3403fa78eb991913db1fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } } }
I20260807 03:14:28.565886 9227 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:28.575625 9234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 03:14:28.577028 9232 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260807 03:14:28.589243 8984 tablet.cc:2369] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 03:14:28.593871 9234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 03:14:28.596529 9200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.576389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.17.31.61:42649)
0807 03:14:28.576648 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.576657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.576740 (+ 83us) server_negotiation.cc:407] Connection header received
0807 03:14:28.577045 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.577051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.577147 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.577235 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.580583 (+ 3348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.581926 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.583872 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.584552 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.584650 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.588765 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:28.588775 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:28.588784 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:28.588837 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:28.592908 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.593846 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.593855 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.593861 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.594069 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.594632 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.594637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.594640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.595033 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:28.595325 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:28.595914 (+ 589us) server_negotiation.cc:299] Negotiation successful
0807 03:14:28.596161 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260807 03:14:28.582119 8963 master_runner.cc:186] Error getting master registration for 127.17.31.61:42649: OK, instance_id { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" instance_seqno: 1786072467881491 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 03:14:28.599855 8963 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 03:14:28.607501 9164 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054 [term 1 FOLLOWER]: Refusing update from remote peer d06d4f1044b3403fa78eb991913db1fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:14:28.608492 9216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:14:28.610224 9112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } attempt: 5
I20260807 03:14:28.619585 9199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.577385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44658 (local address 127.17.31.62:36269)
0807 03:14:28.577560 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.577565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.591549 (+ 13984us) server_negotiation.cc:407] Connection header received
0807 03:14:28.591901 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.591908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.591991 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.592081 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.598658 (+ 6577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.599865 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.603591 (+ 3726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.604273 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.604375 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.612348 (+ 7973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:28.612355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:28.612362 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:28.612402 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:28.616297 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.617152 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.617157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.617161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.617315 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.617767 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.617773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.617777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.618152 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:28.618417 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:28.618975 (+ 558us) server_negotiation.cc:299] Negotiation successful
0807 03:14:28.619203 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 03:14:28.629047 9196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.590837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.17.31.60:44965)
0807 03:14:28.591036 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.591047 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.598880 (+ 7833us) server_negotiation.cc:407] Connection header received
0807 03:14:28.599180 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.599188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.599268 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.599335 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.601184 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.602369 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.604416 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.605056 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.605171 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.609438 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:28.609445 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:28.609450 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:28.609490 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:28.623026 (+ 13536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.624623 (+ 1597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.624628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.624632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.624901 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.625728 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.625736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.625740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.626451 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:28.626847 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:28.627940 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0807 03:14:28.628345 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Aug 07 03:14:28 dist-test-slave-tm1g krb5kdc[8949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786072467, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.31.61@KRBTEST.COM for kudu/127.17.31.60@KRBTEST.COM
I20260807 03:14:28.641755 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.610712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.31.60:44965 (local address 127.0.0.1:48108)
0807 03:14:28.610888 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0807 03:14:28.610910 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 03:14:28.611006 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 03:14:28.623173 (+ 12167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 03:14:28.623178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 03:14:28.623685 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0807 03:14:28.623879 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:28.623892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.625760 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.625768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:28.627197 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 03:14:28.627212 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.628334 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.628341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 03:14:28.628584 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.629511 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 03:14:28.629532 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 03:14:28.632721 (+ 3189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 03:14:28.636859 (+ 4138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:28.636867 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:28.636874 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:28.637691 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:28.638151 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 03:14:28.638184 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 03:14:28.638188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 03:14:28.638494 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 03:14:28.639773 (+ 1279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 03:14:28.639778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 03:14:28.640117 (+ 339us) client_negotiation.cc:769] Sending connection context
0807 03:14:28.640561 (+ 444us) client_negotiation.cc:240] Negotiation successful
0807 03:14:28.641030 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 03:14:28.641669 9202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d06d4f1044b3403fa78eb991913db1fc. Latest consensus state: current_term: 1 leader_uuid: "d06d4f1044b3403fa78eb991913db1fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } } }
I20260807 03:14:28.641970 9202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:14:28.643935 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:28.612589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48108 (local address 127.17.31.60:44965)
0807 03:14:28.622811 (+ 10222us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.622818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.622847 (+ 29us) server_negotiation.cc:407] Connection header received
0807 03:14:28.622900 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.622905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.622988 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.624023 (+ 1035us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.624312 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.625642 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.627436 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:28.628194 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:28.628310 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:28.632943 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 03:14:28.632950 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 03:14:28.632956 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 03:14:28.632994 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 03:14:28.636640 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.637850 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.637858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.637883 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.638038 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 03:14:28.638751 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 03:14:28.638762 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 03:14:28.638766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 03:14:28.639519 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 03:14:28.642337 (+ 2818us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:28.642563 (+ 226us) server_negotiation.cc:299] Negotiation successful
0807 03:14:28.643032 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10071,"thread_start_us":9884,"threads_started":1}
I20260807 03:14:28.648789 9239 mvcc.cc:205] Tried to move back new op lower bound from 7315752831400763392 to 7315752831218630656. Current Snapshot: MvccSnapshot[applied={T|T < 7315752831400763392}]
I20260807 03:14:28.650921 9112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7: Bootstrap starting.
I20260807 03:14:28.657907 9112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7: Neither blocks nor log segments found. Creating new log.
I20260807 03:14:28.658700 9202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d06d4f1044b3403fa78eb991913db1fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } } }
I20260807 03:14:28.658936 9202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:14:28.659973 9112 log.cc:824] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7: Log is configured to *not* fsync() on all Append() calls
I20260807 03:14:28.661806 9216 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d06d4f1044b3403fa78eb991913db1fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } } }
I20260807 03:14:28.662134 9216 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:28.663686 9234 catalog_manager.cc:1347] Generated new cluster ID: 5ccff022f14f497597bfb9375fac68f6
I20260807 03:14:28.663779 9234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 03:14:28.683452 9112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7: No bootstrap required, opened a new log
I20260807 03:14:28.699668 9112 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } }
I20260807 03:14:28.700238 9112 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf7cde467b534aaf9e4c5e4e9f0a86b7, State: Initialized, Role: FOLLOWER
I20260807 03:14:28.701054 9112 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } }
I20260807 03:14:28.716717 9247 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } } }
I20260807 03:14:28.717136 9247 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:14:28.719099 9085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d06d4f1044b3403fa78eb991913db1fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7"
I20260807 03:14:28.719506 9084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 03:14:28.719827 9112 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 03:14:28.721653 9234 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 03:14:28.723315 9084 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7 [term 1 FOLLOWER]: Refusing update from remote peer d06d4f1044b3403fa78eb991913db1fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 03:14:28.723470 9234 catalog_manager.cc:1504] Loading token signing keys...
I20260807 03:14:28.725049 9216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 03:14:28.727550 9085 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d06d4f1044b3403fa78eb991913db1fc in term 1.
I20260807 03:14:28.746915 9234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc: Generated new TSK 0
I20260807 03:14:28.747853 9234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 03:14:28.771574 9247 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d06d4f1044b3403fa78eb991913db1fc. Latest consensus state: current_term: 1 leader_uuid: "d06d4f1044b3403fa78eb991913db1fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } } }
I20260807 03:14:28.772009 9247 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:14:28.772965 9227 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d06d4f1044b3403fa78eb991913db1fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } } }
I20260807 03:14:28.773233 9227 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06d4f1044b3403fa78eb991913db1fc [sys.catalog]: This master's current role is: LEADER
I20260807 03:14:28.771528 9255 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d06d4f1044b3403fa78eb991913db1fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d06d4f1044b3403fa78eb991913db1fc" member_type: VOTER last_known_addr { host: "127.17.31.62" port: 36269 } } peers { permanent_uuid: "cf7cde467b534aaf9e4c5e4e9f0a86b7" member_type: VOTER last_known_addr { host: "127.17.31.61" port: 42649 } } peers { permanent_uuid: "6403fe944af544faa18018989027b054" member_type: VOTER last_known_addr { host: "127.17.31.60" port: 44965 } } }
I20260807 03:14:28.773931 9255 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 03:14:28.801476 9263 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260807 03:14:28.839310 9200 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:28.836125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.17.31.61:42649)
0807 03:14:28.836366 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.836375 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.836913 (+ 538us) server_negotiation.cc:407] Connection header received
0807 03:14:28.837213 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.837219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.837343 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.837432 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.838952 (+ 1520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54874: BlockingRecv error: recv got EOF from 127.0.0.1:54874 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260807 03:14:28.842433 9199 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:28.837053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44666 (local address 127.17.31.62:36269)
0807 03:14:28.837241 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.837247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.840292 (+ 3045us) server_negotiation.cc:407] Connection header received
0807 03:14:28.840600 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.840609 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.840723 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.840812 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.842037 (+ 1225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44666: BlockingRecv error: recv got EOF from 127.0.0.1:44666 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260807 03:14:28.845618 9240 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:28.843858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48110 (local address 127.17.31.60:44965)
0807 03:14:28.844047 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.844052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.844264 (+ 212us) server_negotiation.cc:407] Connection header received
0807 03:14:28.844444 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.844449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.844540 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.844642 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.845444 (+ 802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48110: BlockingRecv error: recv got EOF from 127.0.0.1:48110 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260807 03:14:28.851922 9200 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:28.849751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.17.31.61:42649)
0807 03:14:28.849974 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.849981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.850487 (+ 506us) server_negotiation.cc:407] Connection header received
0807 03:14:28.850725 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.850735 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.850834 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.850941 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.851709 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54882: BlockingRecv error: recv got EOF from 127.0.0.1:54882 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260807 03:14:28.860225 9200 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:28.856124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54890 (local address 127.17.31.61:42649)
0807 03:14:28.856309 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.856316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.858841 (+ 2525us) server_negotiation.cc:407] Connection header received
0807 03:14:28.859022 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.859027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.859100 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.859200 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.859989 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54890: BlockingRecv error: recv got EOF from 127.0.0.1:54890 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260807 03:14:28.867019 9200 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:28.865063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54892 (local address 127.17.31.61:42649)
0807 03:14:28.865242 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.865250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.865762 (+ 512us) server_negotiation.cc:407] Connection header received
0807 03:14:28.865946 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.865952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.866023 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.866121 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.866822 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54892: BlockingRecv error: recv got EOF from 127.0.0.1:54892 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260807 03:14:28.876384 9200 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:28.874497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54906 (local address 127.17.31.61:42649)
0807 03:14:28.874723 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.874730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.875186 (+ 456us) server_negotiation.cc:407] Connection header received
0807 03:14:28.875374 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.875379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.875464 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.875563 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.876163 (+ 600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54906: BlockingRecv error: recv got EOF from 127.0.0.1:54906 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260807 03:14:28.904414 9200 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:28.887564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.17.31.61:42649)
0807 03:14:28.887727 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.887734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.902875 (+ 15141us) server_negotiation.cc:407] Connection header received
0807 03:14:28.903070 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.903077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.903165 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.903274 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.904197 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54918: BlockingRecv error: recv got EOF from 127.0.0.1:54918 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260807 03:14:28.927183 9200 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:28.924573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.17.31.61:42649)
0807 03:14:28.924764 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.924771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.925346 (+ 575us) server_negotiation.cc:407] Connection header received
0807 03:14:28.925647 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.925654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.925743 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.925813 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.926875 (+ 1062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54932: BlockingRecv error: recv got EOF from 127.0.0.1:54932 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260807 03:14:28.966837 9200 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:28.964142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.17.31.61:42649)
0807 03:14:28.964369 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:28.964378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:28.964945 (+ 567us) server_negotiation.cc:407] Connection header received
0807 03:14:28.965267 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:28.965277 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:28.965412 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:28.965492 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:28.966574 (+ 1082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54942: BlockingRecv error: recv got EOF from 127.0.0.1:54942 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260807 03:14:29.049813 9200 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:29.047163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54956 (local address 127.17.31.61:42649)
0807 03:14:29.047357 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:29.047364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:29.048290 (+ 926us) server_negotiation.cc:407] Connection header received
0807 03:14:29.048552 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:29.048558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:29.048653 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:29.048769 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:29.049562 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54956: BlockingRecv error: recv got EOF from 127.0.0.1:54956 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260807 03:14:29.184605 9200 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:29.182369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54958 (local address 127.17.31.61:42649)
0807 03:14:29.182642 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:29.182651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:29.183191 (+ 540us) server_negotiation.cc:407] Connection header received
0807 03:14:29.183421 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:29.183425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:29.183498 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:29.183604 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:29.184374 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54958: BlockingRecv error: recv got EOF from 127.0.0.1:54958 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260807 03:14:29.449975 9200 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 03:14:29.448111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.17.31.61:42649)
0807 03:14:29.448302 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:29.448306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:29.448582 (+ 276us) server_negotiation.cc:407] Connection header received
0807 03:14:29.448763 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:29.448767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:29.448841 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:29.448952 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 03:14:29.449709 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54970: BlockingRecv error: recv got EOF from 127.0.0.1:54970 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 03:14:29.519646 9213 catalog_manager.cc:1259] Loaded cluster ID: 5ccff022f14f497597bfb9375fac68f6
I20260807 03:14:29.519747 9213 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054: loading cluster ID for follower catalog manager: success
I20260807 03:14:29.522400 9213 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054: acquiring CA information for follower catalog manager: success
I20260807 03:14:29.524058 9213 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6403fe944af544faa18018989027b054: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 03:14:29.807101 9263 catalog_manager.cc:1259] Loaded cluster ID: 5ccff022f14f497597bfb9375fac68f6
I20260807 03:14:29.807303 9263 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7: loading cluster ID for follower catalog manager: success
I20260807 03:14:29.811087 9263 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7: acquiring CA information for follower catalog manager: success
I20260807 03:14:29.813500 9263 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cf7cde467b534aaf9e4c5e4e9f0a86b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 03:14:29.972596 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:29.965649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54980 (local address 127.17.31.61:42649)
0807 03:14:29.966528 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:29.966534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:29.966557 (+ 23us) server_negotiation.cc:407] Connection header received
0807 03:14:29.966639 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:29.966645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:29.966829 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:29.966972 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:29.967399 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:29.968668 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:29.969937 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:29.970631 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:29.970744 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:29.971217 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:29.971955 (+ 738us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:29.972111 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:29.972170 (+ 59us) server_negotiation.cc:299] Negotiation successful
0807 03:14:29.972262 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":574,"threads_started":1}
I20260807 03:14:29.985966 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:29.979448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48118 (local address 127.17.31.60:44965)
0807 03:14:29.980129 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:29.980133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:29.980148 (+ 15us) server_negotiation.cc:407] Connection header received
0807 03:14:29.980241 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:29.980244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:29.980383 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:29.980511 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:29.981070 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:29.981807 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:29.983755 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:29.984157 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:29.984249 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:29.984806 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:29.985342 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:29.985449 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:29.985582 (+ 133us) server_negotiation.cc:299] Negotiation successful
0807 03:14:29.985647 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":484,"threads_started":1}
I20260807 03:14:29.999251 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:29.992118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54984 (local address 127.17.31.61:42649)
0807 03:14:29.992279 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:29.992284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:29.992797 (+ 513us) server_negotiation.cc:407] Connection header received
0807 03:14:29.993053 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:29.993060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:29.993147 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:29.993204 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:29.993925 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:29.995043 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:29.997129 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:29.997750 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:29.997836 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:29.998486 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:29.998601 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:29.998660 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:29.998901 (+ 241us) server_negotiation.cc:299] Negotiation successful
0807 03:14:29.998981 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 03:14:30.008059 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:29.999284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.17.31.60:44965)
0807 03:14:29.999522 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:29.999527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:30.001380 (+ 1853us) server_negotiation.cc:407] Connection header received
0807 03:14:30.001632 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:30.001637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:30.001722 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:30.001783 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:30.002423 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:30.003520 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:30.005745 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:30.006409 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:30.006515 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:30.007245 (+ 730us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:30.007399 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:30.007474 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:30.007700 (+ 226us) server_negotiation.cc:299] Negotiation successful
0807 03:14:30.007781 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260807 03:14:30.009086 9343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 03:14:29.992101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.17.31.62:36269)
0807 03:14:29.998670 (+ 6569us) server_negotiation.cc:206] Beginning negotiation
0807 03:14:29.998678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 03:14:29.999255 (+ 577us) server_negotiation.cc:407] Connection header received
0807 03:14:29.999382 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 03:14:29.999390 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 03:14:29.999591 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 03:14:29.999776 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 03:14:30.000672 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:30.002709 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:30.005652 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 03:14:30.006656 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 03:14:30.006800 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 03:14:30.007831 (+ 1031us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 03:14:30.008277 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 03:14:30.008401 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 03:14:30.008651 (+ 250us) server_negotiation.cc:299] Negotiation successful
0807 03:14:30.008793 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6443,"thread_start_us":6346,"threads_started":1}
I20260807 03:14:30.014703 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 8963
I20260807 03:14:30.015671 9027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:14:30.125571 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 8963
I20260807 03:14:30.155408 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 9040
I20260807 03:14:30.156572 9106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:14:30.271073 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 9040
I20260807 03:14:30.291920 17532 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu pid 9120
I20260807 03:14:30.293053 9187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 03:14:30.398993 17532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko9xIn4/build/asan/bin/kudu with pid 9120
2026-08-07T03:14:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4902 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9309 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (363344 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST